How to Recreate Reddit with Dgraph and Vue.js
Dgraph is able to handle robust data sets, even those from applications built designed to use more traditional relational databases. With just some minor adjustments to your thinking you’ll see how Dgraph and its GraphQL+- syntax can be used to work with large data sets from popular real-world applications.
In this tutorial we’ll walk step-by-step through the process if creating a Reddit-style front end application using massive, real-world Reddit data dumps imported into Dgraph. We’ll explore how Dgraph efficiently manages large data sets, even while running within relatively humble Docker environments. We’ll also see how, with just a few minor adjustments to our application logic, we can use Dgraph to query and mutate data that wasn’t designed to work with a graph database like Dgraph in the first place. We’ll see how multiple types of objects can share the same fields (i.e. predicates) within Dgraph, and yet the power of GraphQL+- queries will allow us to differentiate between record types in whatever manner is best suited to our application.
In addition to Dgraph for our database we’ll also be using the Vue.js framework to create the front end application. This lets us build out a responsive and modern version of Reddit that can easily be modified to include addition functionality. Vue provides a number of advantages over other popular libraries by allowing us to combine HTML, CSS, and JavaScript for a given component within a single file, which dramatically improves readability and separation of concerns.
You are encouraged to follow along with the tutorial step-by-step, but if you want to reference to full code set feel free to check out the repository at any time. With that, let’s jump right into it!
Create a Vue CLI Project
Start by globally installing @vue/cli
.
yarn global add @vue/cli
Create a new Vue project. In this tutorial our project will be named dgraph-reddit
, but you can call your project anything you’d like.
vue create dgraph-reddit
Choose to Manually select features
and then choose the following options:
- Please pick a preset: Manually select features
- Check the features needed for your project: Babel, TS, Router, Vuex, CSS Pre-processors, Linter
- Use class-style component syntax? Yes
- Use Babel alongside TypeScript for auto-detected polyfills? Yes
- Use history mode for router? (Requires proper server setup for index fallback in production) Yes
- Pick a CSS pre-processor (PostCSS, Autoprefixer and CSS Modules are supported by default): Sass/SCSS (with dart-sass)
- Pick a linter / formatter config: TSLint
- Pick additional lint features: Lint on save
- Where do you prefer placing config for Babel, PostCSS, ESLint, etc.? In dedicated config files
Wait a few moments for the installation to complete then navigate to the project directory and execute yarn serve
to see that your app is up and running. By default, the dev server is available at http://localhost:8080/
, but vue-cli-service
may choose a different port if it detects 8080
is in use.
$ cd dgraph-reddit
$ yarn serve
App running at:
- Local: http://localhost:8080/
- Network: http://10.0.75.1:8080/
You can opt to keep the dev server running once development begins, but for now it’s best to terminate it (Ctrl/Cmd + C
).
Install Node Packages
Let’s also take a moment to install some additional dependencies we’ll need to get started, particularly since we’ll be using TypeScript.
yarn add dgraph-js-http gulp gulp-typescript
yarn add -D @babel/polyfill @types/gulp @types/node ts-node cli-progress @types/cli-progress
Prettier & TSLint
We’ll be using Prettier and TSLint for our linting needs throughout this guide, but feel free to use whatever setup you prefer. If you wish to use the same solution, you’ll want to configure your editor to execute Prettier/linting upon save. That’s beyond the scope of this article, but check out the prettier-vscode and Prettier + WebStorm documentation for more details on using some popular editors.
I also recommend adding some additional rules to the project tslint.json
file to disable the no-console
and trailing-comma
rules.
{
"defaultSeverity": "warning",
"extends": ["tslint:recommended"],
"linterOptions": {
"exclude": ["node_modules/**"]
},
"rules": {
"quotemark": [true, "single"],
"indent": [true, "spaces", 2],
"interface-name": false,
"ordered-imports": false,
"object-literal-sort-keys": false,
"no-consecutive-blank-lines": false,
"no-console": false,
"trailing-comma": false
}
}
I’ve also created a .prettierrc
file in the project root directory with the following content.
{
"semi": true,
"singleQuote": true
}
Connecting to Dgraph
With our project configured we’re ready to get into playing around with Dgraph and adding some data that our app will use.
Prerequisites
Make sure you have a Dgraph installation available for use explicitly for this project, as you don’t want to risk your existing data. Installing Dgraph takes just a few minutes and can be easily accomplished on the local system or within a Docker container. Check out the official Get Started documentation for details on that!
Throughout the rest of this tutorial we’ll assume you’re using a local Dgraph installation at the default URL of http://localhost:8080
, so please change the address to match your local install if needed.
Connecting to Dgraph Alpha
To add our Reddit data to Dgraph we’ll need to establish a connection to Dgraph’s Alpha server which hosts the predicates and indices Dgraph relies on to represent data. Since we’re creating a front-end Vue application we’ll be using the dgraph-js-http
library which provides some helper methods for connecting to Dgraph and executing transactions.
Let’s start by creating a new directory at
src/dgraph
to hold all our Dgraph code.mkdir src/dgraph && cd src/dgraph
Create a new file call
DgraphAdapter.ts
.touch DgraphAdapter.ts
The
DgraphAdapter
class where we’ll establish a connection between our app and Dgraph through thedgraph-js-http
library. Let’s start by importing theDgraphClient
andDgraphClientStub
fromdgraph-js-http
, which we’ll use to establish a Dgraph connection. We’ll also create theclient
andclientStub
properties, which will hold non-nullable instances of theDgraphClient
andDgraphClientStub
objects. Theaddress
property is just the string pointing to our Dgraph Alpha server endpoint.import { DgraphClient, DgraphClientStub } from 'dgraph-js-http'; export class DgraphAdapter { public address = 'http://localhost:8080'; protected client: NonNullable<DgraphClient>; protected clientStub: NonNullable<DgraphClientStub>; }
Let’s next add the
constructor
method, which will accept an optionaladdress?: string
argument.constructor(address?: string) { if (address) { this.address = address; } this.clientStub = new DgraphClientStub(this.address); this.client = new DgraphClient(this.clientStub); }
This allows us to either use the default Dgraph Alpha address or override the default with our own address during invocation. Otherwise, calling the constructor just creates a new instance of the
DgraphClient
using theDgraphClientStub
and sets the properties.Your
DgraphAdapter.ts
should look like the following.import { DgraphClient, DgraphClientStub } from 'dgraph-js-http'; export class DgraphAdapter { public address = 'http://localhost:8080'; protected client: NonNullable<DgraphClient>; protected clientStub: NonNullable<DgraphClientStub>; constructor(address?: string) { if (address) { this.address = address; } this.clientStub = new DgraphClientStub(this.address); this.client = new DgraphClient(this.clientStub); } }
Let’s leave this for now and quickly setup a way for us to execute arbitrary code that isn’t tied directly to our Vue application. We’ll be using
Gulp
to handle such tasks.
Setting Up Gulp
Like the rest of the project we’ll be using TypeScript and a number of ES6+ JavaScript features within Gulp. However, the default tsconfig.json
that the Vue CLI
created for us is configured to use the esnext
module, which will cause an error when trying to execute Gulp commands. However, our Vue application has to use the esnext
module code generation to use all the advanced features it has. The problem is that we effectively need to specify two different sets of TypeScript configurations depending on the execution stack (Vue vs Gulp), so how can we do it? The first inclination might be to pass options to the Node commands we’re invoking (such as gulp
or yarn serve
), but that runs into complications when we’re trying to invoke TypeScript compilation first. In particular, most libraries are configured to look for a tsconfig.json
file in the root directory and overriding that project location is difficult depending on the order of operations.
To solve this issue we need to make a copy the existing
tsconfig.json
since it is pre-configured for our Vue application. We’ll call ittsconfig.vue.json
.cp tsconfig.json tsconfig.vue.json
Now we need to edit the base
tsconfig.json
file and change themodule
property tocommonjs
to ensure it remains compatible with Gulp when it is initially registered withts-node/register
upon execution.{ "compilerOptions": { "target": "esnext", "module": "commonjs" // ... } }
The
ts-node
package will find the defaulttsconfig.json
and use it for our Gulp commands that are separate from our Vue application. However, if we run our Vue app again we see a compilation error is now occurring.$ yarn serve WARNING Compiled with 2 warnings warning in ./src/components/HelloWorld.vue?vue&type=script&lang=ts& "export 'default' (imported as 'mod') was not found in '-!../../node_modules/cache-loader/dist/cjs.js??ref--13-0!../../node_modules/babel-loader/lib/index.js!../../node_modules/ts-loader/index.js??ref--13-2!../../node_modules/cache-loader/dist/cjs.js??ref--0-0!../../node_modules/vue-loader/lib/index.js??vue-loader-options!./HelloWorld.vue?vue&type=script&lang=ts&' warning in ./src/views/Home.vue?vue&type=script&lang=ts& "export 'default' (imported as 'mod') was not found in '-!../../node_modules/cache-loader/dist/cjs.js??ref--13-0!../../node_modules/babel-loader/lib/index.js!../../node_modules/ts-loader/index.js??ref--13-2!../../node_modules/cache-loader/dist/cjs.js??ref--0-0!../../node_modules/vue-loader/lib/index.js??vue-loader-options!./Home.vue?vue&type=script&lang=ts&'
Even though these are just warnings and compilation completes, it results in a non-functional Vue application.
As mentioned above, these errors are a result of Vue using the default
tsconfig.json
for TypeScript compilation, which is no longer set to use theesnext
module for code generation. To fix this we need to tell Vue where to find thetsconfig.vue.json
configuration when transpiling. Vue CLI apps are built on top of Webpack by default, so we can modify the Webpackts
rule so thets-loader
knows where to find the configuration file we want it to use.Create a
vue.config.js
file in the project root directory and add the following to it.module.exports = { chainWebpack: config => { config.module .rule('ts') .use('ts-loader') .loader('ts-loader') .tap(options => { options.configFile = 'tsconfig.vue.json'; return options; }); } };
Webpack configuration is essentially defined as a series of rules based on file types. Each rule has its own options that tell Webpack what to do with matching files. In the
vue.config.js
above we’re using thechainWebpack
property to add a configuration rule forts
files to Webpack. It uses thets-loader
and then calls thetap()
method (provided by theTapable
core utility) to inject a additional build step into the Webpack chain. As you can see, we’re merely modifying theconfigFile
location sots-loader
will usetsconfig.vue.json
, instead of looking for the defaulttsconfig.json
file.With that configuration added let’s try running the Vue dev server again.
$ yarn serve App running at: Local: http://localhost:8080/ Network: http://10.0.75.1:8080/
Sure enough, that does the trick and our Vue app is able to compile without any errors.
Tip The use of Webpack is beyond the scope of this tutorial, but it’s a powerful and popular bundler that warrants learning about. Check out the official documentation for more details.
Dropping Dgraph Data
With our Gulp ready to go we can start implementing some business logic into DgraphAdapter
so it can establish a connection with Dgraph and perform some transactions.
The first thing we want to do is clean out our Dgraph database so we’re starting fresh, so let’s open the
src/dgraph/DgraphAdapter.ts
file and add a newdropAll()
method to the class.export class DgraphAdapter { // ... public async dropAll(): Promise<boolean> { try { const payload: any = await this.client.alter({ dropAll: true }); if (payload.data.code && payload.data.code === 'Success') { console.info(`All Dgraph data dropped.`); return true; } else { console.info(`Dgraph data drop failed.`); return false; } } catch (error) { console.error(`Dgraph data drop failed, error: %s`, error); return false; } } // ... }
This method is rather simple. We’re invoking the
alter()
method of theDgraphClient
instance and passing an argument object of{ dropAll: true}
, which sends a request to Dgraph to drop all data. We then check the payload result for adata.code
property indicating a success, otherwise we assume failure.Open
gulpfile.ts
and let’s add thedb:drop
task with the following code.import gulp from 'gulp'; import { DgraphAdapter } from './src/dgraph/DgraphAdapter'; gulp.task('db:drop', () => { try { return new DgraphAdapter().dropAll(); } catch (error) { throw error; } });
Gulp allows us to create tasks that can then be executed individually, or in combined series with other Gulp tasks. Here we just create an anonymous function that is executed when the
db:drop
Gulp task is invoked. In this function we return the result of ourDgraphAdapter
instance’sdropAll()
method.Tip Gulp tasks expect the result of their function call to be aPromise
, which is what Gulp uses to determine when a given task has completed. In this case, theDgraphAdapter.dropAll()
method isasync
, which returns aPromise
by default (though, since we’re using TypeScript, we also explicitly specified the return type ofPromise<boolean>
). Check out the official website for more information on using Gulp and creating custom tasks.Save the
gulpfile.ts
changes and execute thedb:drop
Gulp task with the following command.$ gulp db:drop [21:56:56] Requiring external module ts-node/register [21:56:58] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [21:56:58] Starting 'db:drop'... All Dgraph data dropped. [21:56:58] Finished 'db:drop' after 438 ms
If all goes well you should see Gulp’s output similar to the above, indicating that the
db:drop
task executed and that Dgraph data was dropped. You can open the Dgraph Ratel web UI to confirm this is the case. Just navigate to the Schema tab and you should see only the default schema predicate entries (which begin withdgraph.
prefixes).Info If you receive a connection error indicating Dgraph is unavailable, double-check that theDgraphAdapter.address
property matches the Dgraph Alpha server URL in which you have Dgraph installed and running at.
Adding Environment Variables
Speaking of the Dgraph Alpha server address, it’s rather inconvenient to have to manually edit the static URL within the DgraphAdapter.address
property anytime the Dgraph server changes, so let’s remedy this (and many future configuration headaches) by quickly adding environment variable support to our app.
Start by installing the dotenv package from NPM.
yarn add dotenv @types/dotenv
This package allows us to create
.env
files in the project root directory that initialize environment variables.Create a new
.env
file in the project root directory.Add the following entry to the
.env
file, changing the URL to match your particular setup.DGRAPH_ALPHA_URL=http://localhost:8080
Let’s add support for this new environment variable in
src/dgraph/DgraphAdapter
. Open that file and defaultaddress
property value toprocess.env.DGRAPH_ALPHA_URL
.export class DgraphAdapter { public address = process.env.DGRAPH_ALPHA_URL; // ... }
Now we need to bootstrap the
dotenv
configuration by calling itsconfig()
method, ideally before the rest of our application executes. Since we have Gulp tasks that are run separately from our Vue app we need to add the bootstrap code at the top of two files:gulpfile.ts
andsrc/main.ts
.import dotenv from 'dotenv'; dotenv.config(); import gulp from 'gulp'; import { DgraphAdapter } from './src/dgraph/DgraphAdapter'; gulp.task('db:drop', () => { try { return new DgraphAdapter().dropAll(); } catch (error) { throw error; } });
import dotenv from 'dotenv'; dotenv.config(); import Vue from 'vue'; import App from './App.vue'; import router from './router'; import store from './store'; Vue.config.productionTip = false; new Vue({ router, store, render: h => h(App) }).$mount('#app');
Any values we add to the
.env
file will now be available in theprocess.env
object throughout our app.With
DgraphAdapter
updated to use our env variable let’s test out that ourgulp db:drop
command works.$ gulp db:drop [22:08:28] Requiring external module ts-node/register [22:08:29] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [22:08:29] Starting 'db:drop'... All Dgraph data dropped. [22:08:30] Finished 'db:drop' after 424 ms
It should work just as well as before, but now we’re not using a static Dgraph Alpha server URL. This will come in handy throughout the app creation process.
Altering the Dgraph Schema
Now that we can drop data let’s alter the schema. As we’ll see shortly, it isn’t technically necessary to initially generate a schema for Dgraph data since Dgraph will generate predicates for us on the fly. However, we’ll probably want to specify at least some predicate schema later on so we can enable additional Dgraph features such as indexing.
Start by opening
src/dgraph/DgraphAdapter.ts
and adding the newalterSchema()
method.public async alterSchema(schema: string): Promise<boolean> { try { const payload: any = await this.client.alter({ schema }); if (payload.data.code && payload.data.code === 'Success') { console.info(`Dgraph schema altered.`); return true; } else { console.info(`Dgraph schema alteration failed.`); return false; } } catch (error) { console.error(`Dgraph schema alteration failed, error: %s`, error); return false; } }
This method accepts a new schema
string
and performs an alteration using said schema.Now go back to the
gulpfile.ts
and let’s add a newdb:schema:alter
task.gulp.task('db:schema:alter', () => { try { return new DgraphAdapter().alterSchema(DGRAPH_SCHEMA); } catch (error) { throw error; } });
This task passes an undefined
DGRAPH_SCHEMA
value, so let’s initialize that at the top of thegulpfile.ts
.const DGRAPH_SCHEMA = ` createdAt: dateTime @index(hour) . description: string @index(fulltext) @count . email: string @index(exact) @upsert . name: string @index(hash) @count . `;
A Dgraph schema defines the data type for each given predicate using a
predicate: type [@directive(s)]
format. Here we’re adding 4 predicates, each with anindex
directive. The specifics don’t matter for the moment, as we’re just using this to test that we’re able to alter the schema, so we can update this later as we build out the app.Our Gulp task is ready so let’s try it out.
$ gulp db:schema:alter [01:23:26] Requiring external module ts-node/register [01:23:27] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [01:23:27] Starting 'db:schema:alter'... Dgraph schema altered. [01:23:27] Finished 'db:schema:alter' after 77 ms
Everything looks good from the console. We can confirm the schema was altered by checking the Schema tab of the Ratel web UI. We should now see the four new predicates, in addition to the baseline
dgraph.
predicates.
Adding Dgraph Mutations
The next milestone to add to our app is the ability to perform mutations within Dgraph. This will allow our app to add or remove data using a simple JSON format.
Open
src/dgraph/DgraphAdapter.ts
and add aMutationTypes
enum at the top, which we’ll use to help us differentiate which type of mutation we’re performing.export enum MutationTypes { DeleteJson, SetJson }
Now let’s create the
DgraphAdapter.mutate()
method, which will accept arequest
payload, perform the type of mutation we need, and potentially return anArray
ofuid
values that were created.public async mutate<T>({ request, mutationType = MutationTypes.SetJson, commitNow = false }: { request: any; mutationType?: MutationTypes; commitNow?: boolean; }): Promise<string[]> { if (request === undefined) { throw Error( `DgraphAdapter.mutate error, payload undefined for request: ${request}` ); } const transaction = this.client.newTxn(); let uids: string[] = []; try { const mutation: Mutation = {}; mutation.commitNow = commitNow; if (mutationType === MutationTypes.SetJson) { mutation.setJson = request; } else if (mutationType === MutationTypes.DeleteJson) { mutation.deleteJson = request; } const assigned: Assigned = await transaction.mutate(mutation); if (!commitNow) { await transaction.commit(); } uids = Object.entries(assigned.data.uids).map(([key, uid]) => uid); } catch (e) { console.error( 'DgraphAdapter.mutate, request: %o, mutationType: %o, error: %o', request, mutationType, e ); } finally { await transaction.discard(); } return uids; }
The
mutate
method accepts an object argument with arequest
property which contains the JSON we’re mutating. We create a new transaction and instantiate a newMutation
object that comes fromdgraph-js-http
. This object also shows that it can accept either asetJson
ordeleteJson
property, which informsdgraph-js-http
whether we’re adding or removing data. The optionalcommitNow
property is used to tell Dgraph whether to immediately commit the mutation or not, so we use the passedcommitNow
parameter to alter that behavior.After committing the mutation we
.map
the results from theAssigned
object to the returneduids
array.Back in the
gulpfile.ts
let’s add a newdb:mutate:test
task to test our newDgraphAdapter.mutate()
method.gulp.task('db:mutate:test', async () => { try { const request = { createdAt: new Date(), description: 'Hello, this is Alice!', email: 'alice@example.com', name: 'Alice Jones' }; const result = await new DgraphAdapter().mutate({ request }); console.log(result); } catch (error) { throw error; } });
This Gulp task creates a plain object for our request and passes it to the
request
property of themutate()
argument.Finally, execute the
gulp db:mutate:test
command from a terminal.$ gulp db:mutate:test [16:54:15] Requiring external module ts-node/register [16:54:16] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [16:54:16] Starting 'db:mutate:test'... [ '0x4ce7d' ] [16:54:16] Finished 'db:mutate:test' after 53 ms
Sure enough, this works just as expected and the awaited
result
is astring[]
containing the list of newly-generateduid
values in Dgraph. Since we only added a single node, we only receive a singleuid
in return. Regardless, we knowDgraphAdapter.mutate()
is ready to go. Time to move onto querying for existing data!
Querying Dgraph
To perform a Dgraph query we once again need to create a new transaction and pass our query string (and optional arguments) to retrieve some data. We’ll add the query()
method to our DgraphAdapter
class to help with this.
Open
src/dgraph/DgraphAdapter.ts
and add the followingquery
method.public async query<T>(query: string, vars?: object): Promise<any> { const transaction = this.client.newTxn(); let result; try { // Reduce optional vars to string values only. vars = vars ? Object.entries(vars).reduce((accumulator: any, value) => { accumulator[value[0]] = value[1].toString(); return accumulator; }, {}) : vars; const response: Response = vars ? await transaction.queryWithVars(query, vars) : await transaction.query(query); result = response.data; } catch (error) { console.error('DgraphAdapter.query, query: %o, error: %o', query, error); } finally { await transaction.discard(); } return result; }
Similar to our other
DgraphAdapter
helper methods,query()
creates a transaction, then passes ourquery
string to theTxn.query()
orTxn.queryWithVars()
method, depending if we’ve provided optionalvars
arguments. We extract the resulting data and return that result.Let’s add a simple Gulp task to help test out our queries as well. Inside
gulpfile.ts
add thedb:query:test
task seen below.gulp.task('db:query:test', async () => { try { const query = ` { user(func: eq(email, "alice@example.com")) { uid expand(_all_) { uid expand(_all_) } } } `; const result = await new DgraphAdapter().query(query); console.log(result); } catch (error) { throw error; } });
Here we’re defining our first GraphQL+- query string, so let’s take a moment to break down the components of the query.
user
is a user-defined and completely arbitrary name for the query block we’re defining. Since we’re looking at a user record of sorts, a name ofuser
seems appropriate.func: eq(email, "alice@example.com")
- GraphQL+- functions allow us to filter the results based on specified arguments, such as predicate values. In this case theeq
function accepts a predicate and a value argument, and filters nodes in which that predicate value equals the passed value argument. Therefore, here we’re filtering for nodes where theemail
predicate equalsalice@example.com
.uid
is a special predicate type that is automatically generated and added to every node and uniquely identifies it. While typically represented in base 16 format the underlying value of auid
is auint64
.expand(_all_)
- Theexpand()
function is a special function that can be used to expand all child predicates passed to it. The_all_
keyword is a shortcut that stands in for all predicates, soexpand(_all_)
just tells Dgraph to expand and include every child predicate at that level of the node. As you can see above, such blocks can be chained together, so we can expand within another expansion as often as needed.
Tip Dgraph will ignore extra spacing around the query, so the formatting above is just for readability purposes. That same query could be written in a single line and would function just the same:{user(func:eq(email,"alice@example.com")){uid expand(_all_) {uid expand(_all_)}}}
. You can learn much more about GraphQL+- query syntax in the official documentation.With our new
gulp db:query:test
task setup let’s finally test it out.$ gulp db:query:test [17:18:27] Requiring external module ts-node/register [17:18:28] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [17:18:28] Starting 'db:query:test'... { user: [ { uid: '0x4ce7d', email: 'alice@example.com', description: 'Hello, this is Alice!', createdAt: '2019-04-17T00:15:02.538Z', name: 'Alice Jones' } ] } [17:18:28] Finished 'db:query:test' after 41 ms
Awesome, everything works as expected. We can see that the returned result is a JavaScript
Object
that contains our arbitraryuser
property, which is assigned to an array of objects containing the resulting nodes that matched our query filter. Since our data set only contains a single matching node, only one was returned, but the nature of this data structure allows Dgraph to return arbitrarily-sized data based on the query results.
Reddit Data Dump
We’ll be using data dumps that come directly from Reddit and are provided by the pushshift.io
project, which can be found here. There are a number of subcategories for the data that is available, but this tutorial will be focusing on just two categories: comments and submissions. If you’re familiar with Reddit, every top-level post is a submission, and then each submission can contain many child comments in response to that submission. Therefore, these two types of data will cover the majority of the information seen on the actual Reddit site.
Since these data sets start to get quite huge, we’ll just use a small sampling to popular our initial app data. We’ll grab daily dumps of both comments and submissions from two arbitrary dates of 2018-02-01
and 2018-02-02
.
- RC_2018-02-01: http://files.pushshift.io/reddit/comments/daily/RC_2018-02-01.xz
- RC_2018-02-02: http://files.pushshift.io/reddit/comments/daily/RC_2018-02-02.xz
- RS_2018-02-01: http://files.pushshift.io/reddit/submissions/daily/RS_2018-02-01.xz
- RS_2018-02-02: http://files.pushshift.io/reddit/submissions/daily/RS_2018-02-02.xz
Create a
src/data
directory and navigate into it.mkdir data && cd data
Execute the following curl commands (or whatever downloader you prefer) to grab the submission and comment data sets.
curl -O http://files.pushshift.io/reddit/submissions/daily/RS_2018-02-01.xz && curl -O http://files.pushshift.io/reddit/submissions/daily/RS_2018-02-02.xz curl -O http://files.pushshift.io/reddit/comments/daily/RC_2018-02-01.xz && curl -O http://files.pushshift.io/reddit/comments/daily/RC_2018-02-02.xz
Once downloaded, extract the archived contents with the
unxz *.xz
command, which will unzip all.xz
files in the current directory.unxz *.xz
Info These files are approximately6GB
in total size after decompression, so make sure you have appropriate disk space prior to this step.With the data extracted we can start adding it to Dgraph.
Adding Reddit Data to Dgraph
If you look at the content of one of the RS_
or RC_
Reddit data dump files, you’ll see they are in single-line JSON format. Here’s a submission record example.
{
"archived": false,
"author": "transcribersofreddit",
"author_flair_css_class": null,
"author_flair_text": "Official Bot",
"brand_safe": false,
"contest_mode": false,
"created_utc": 1517443200,
"distinguished": null,
"domain": "reddit.com",
"edited": false,
"gilded": 0,
"hidden": false,
"hide_score": false,
"id": "7ueit6",
"is_crosspostable": false,
"is_reddit_media_domain": false,
"is_self": false,
"is_video": false,
"link_flair_css_class": "unclaimed",
"link_flair_text": "Unclaimed",
"locked": false,
"media": null,
"media_embed": {},
"no_follow": true,
"num_comments": 1,
"num_crossposts": 0,
"over_18": false,
"parent_whitelist_status": null,
"permalink": "/r/TranscribersOfReddit/comments/7ueit6/toomeirlformeirl_image_toomeirlformeirl/",
"pinned": false,
"retrieved_on": 1520467337,
"score": 1,
"secure_media": null,
"secure_media_embed": {},
"selftext": "",
"send_replies": true,
"spoiler": false,
"stickied": false,
"subreddit": "TranscribersOfReddit",
"subreddit_id": "t5_3jqmx",
"subreddit_type": "public",
"suggested_sort": null,
"thumbnail": "default",
"thumbnail_height": 140,
"thumbnail_width": 140,
"title": "TooMeIrlForMeIrl | Image | \"TooMeIrlForMeIrl\"",
"url": "https://reddit.com/r/TooMeIrlForMeIrl/comments/7ueit3/toomeirlformeirl/",
"whitelist_status": null
}
As we already saw above, Dgraph’s GraphQL+- engine can accept JSON data within mutations, so we just need to create a helper function that can efficiently extract this data and push it to a Dgraph mutation. However, since these are rather large files, we cannot simply load and read their content from memory. The solution we’ll be using is Node Streams.
A stream in Node is a rather abstract concept that is used throughout the standard library. Conceptually, a stream is a collection of data similar to an array or object, with one major caveat: Stream data is temporal since it mutates over time. So, in the case of reading the data from our 2+ GB file, we don’t have to read it all at once and hold the data in memory, but we’ll instead use a stream that will slowly trickle the data in over time in chunks, which we’ll then process as needed.
We’ll start by installing the
event-stream
library, which is an extremely popular collection of helper methods for working with Node streams.yarn add event-stream && yarn add -D @types/event-stream
In this case, we’ll be using
event-stream
to split our data by line and format it into JSON objects before we manipulate it with custom code. We’ll also be using [cli-progress
]() to create a progress bar.Next, open the
src/dgraph/DgraphAdapter.ts
file and add the followingmutateFromStream
method.import es from 'event-stream'; import * as CliProgress from 'cli-progress'; // ... public static async mutateFromStream({ stream, batchSize = 50, limit = 150 }: { stream: ReadStream; batchSize?: number; limit?: number; }) { const adapter = new DgraphAdapter(); let batch: any[] = []; let total = 0; const bar = new CliProgress.Bar( { stopOnComplete: true, format: '{bar} {percentage}% | Elapsed: {duration_formatted} | ETA: {eta_formatted} | {value}/{total} records' }, CliProgress.Presets.shades_classic ); // Start progress bar with maximum of limit. bar.start(limit, 0); const syncMutation = async (readStream: ReadStream, event?: string) => { try { // Pause during async. readStream.pause(); // Mutate batch const response = await adapter.mutate({ request: batch }); // Reset batch. batch = []; // Update progress bar. bar.update(total); // Resume after async. readStream.resume(); } catch (error) { // Stop progress bar. bar.stop(); console.log(error); } }; return new Promise((resolve, reject) => { stream .pipe(es.split()) .pipe(es.parse()) .on('data', async function(this: ReadStream, data: any) { // Add data to batch and update total count. batch.push(data); total++; if (total >= limit) { // Close stream if total exceeds limit. this.destroy(); } else if (batch.length === batchSize) { // Synchronously mutate if batch length meets batchSize. await syncMutation(this, 'data'); } }) .on('error', (error: Error) => { // Stop progress bar. bar.stop(); console.log(error); reject(error); }) .on('close', async function(this: ReadStream) { // Synchronously mutate if batch contains any extraneous records. if (batch.length > 0) { await syncMutation(this, 'close'); } // Stop progress bar. bar.stop(); resolve(`Stream closed, processed ${total} out of ${limit} records.`); }); }); }
To see what the
mutateFromStream
method is doing start down at thereturn new Promise()
line, which begins the process of asynchronously reading data from aReadStream
object. As mentioned above, we useevent-stream's
split()
andparse()
methods here to first split our data stream by line (the default), then parse that data into a JSON object.Each chunk of data passed to the
on('data', ...)
event is a regular JavaScript object. Each data object is added to thebatch
array. We then check if the total number of records processed exceeds thelimit
threshold, in which case we destroy the stream immediately. Thelimit
parameter can be used to process a certain number of objects from our data stream, which is useful when dealing with massive data sets like the Reddit dump files. Calling the.destroy()
method of aReadStream
object invokes theend
event (which we aren’t handling) and theclose
event.If the current
batch
size equals thebatchSize
parameter (or if the stream is closing andbatch
still contains data) we invoke thesyncMutation
function and await the result.syncMutation
synchronously pushes thebatch
data to Dgraph via theDgraphAdapter.mutate
method. It’s critical that we explicitly pause the stream whileawaiting
the mutation result, in order to avoid race conditions and prevent building up back pressure (i.e. reading data faster than our Dgraph consumer can process it). Once abatch
of data has been mutated and a response is receieved we reset thebatch
collection andresume
the stream as before.It’s also important to note that the stream always invokes the
close
method at the end of its lifecycle, whether we.destroy()
it for reaching our record limit, or because it simply processed all available stream data. Thus, we perform a final cleanup in casebatch
contains any extra records that weren’t processed within abatchSize
chunk.Now that we’ve got the
mutateFromStream
method we can test it out within a Gulp task. Before we do, however, let’s add the command line argument parsing library minimist.yarn add minimist && yarn add -D @types/minimist
This library provides some convenience for reading and handling additional arguments passed to terminal commands, such as Gulp tasks. This will allow us to dynamically provide arguments for
mutateFromStream
, so we can adjust thebatchSize
andlimit
, for example.In
gulpfile.ts
add the newdb:generate:data
task using the code below.gulp.task('db:generate:data', async () => { try { const args = minimist(process.argv.slice(3), { default: { batchSize: 250, limit: 1000, path: './src/data/RS_2018-02-01' } }); const stream = fs.createReadStream(args.path, { flags: 'r' }); // Include optional command line arguments in options. const result = await DgraphAdapter.mutateFromStream( Object.assign( { stream }, args ) ); console.log(result); } catch (error) { throw error; } });
Here we’re creating a
ReadStream
from the2018-02-01
Reddit submissions data set and then generating anoptions
object that contains ourstream
instance. We also specify thebatchSize
,limit
, andpath
properties, all of which receive default values using theminimist
library. Thepath
property is used to create theReadStream
, while the other arguments are passed toDgraphAdapter.mutateFromStream()
.Let’s test it out by running
gulp db:generate:data
from the command line.$ gulp db:generate:data [21:47:07] Requiring external module ts-node/register [21:47:08] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [21:47:08] Starting 'db:generate:data'... ████████████████████████████████████████ 100% | Elapsed: 3s | ETA: 0s | 5000/5000 records Stream closed, processed 1000 out of 1000 records. [21:47:11] Finished 'db:generate:data' after 3.42 s
The output should look something like the above. During progression you should notice that the record count is properly increasing by the
batch.length
size of250
, which means our code is aggregating anArray
of250
objects read from thedata
stream event prior to performing aDgraphAdapter.mutate()
call. We can also see that we’re correctly processing only thelimit
number of records (1000
, by default) before the stream is destroyed and closed.Let’s also confirm that we can override the default
batchSize
andlimit
parameters by providing matching--arguments
to thegulp db:generate:data
command. Let’s try abatchSize
of123
and alimit
of2000
$ gulp db:generate:data --batchSize 123 --limit 2000 [21:52:07] Requiring external module ts-node/register [21:52:08] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [21:52:08] Starting 'db:generate:data'... ████████████████████████████████████████ 100% | Elapsed: 7s | ETA: 0s | 5000/5000 records Stream closed, processed 2000 out of 2000 records. [21:52:42] Finished 'db:generate:data' after 7.82 s
Awesome, both our argument overrides are working as expected. The last thing to do is confirm that the data is showing up in Dgraph as expected.
Open up the Ratel UI (http://localhost:8000/?latest), navigate to Console > Query and run the following query. We’re explicitly filtering out the, well, explicit content from the query below to ensure this is safe for work, but feel free to remove those filters throughout the tutorial if you want to ensure everything is as accurate as possible.
{ data(func: has(author), first: 10) @filter(eq(over_18, false)) { uid expand(_all_) { uid expand(_all_) } } }
You should see a list of the first
10
submissions that contain anauthor
predicate.
{
data(func: has(author), first: 10) @filter(eq(over_18, false)) {
uid
expand(_all_) {
uid
expand(_all_)
}
}
}
curl http://127.0.0.1:8080/query -XPOST -d '
{
data(func: has(author), first: 10) @filter(eq(over_18, false)) {
uid
expand(_all_) {
uid
expand(_all_)
}
}
}' | python -m json.tool | less
package main
import (
"context"
"flag"
"fmt"
"log"
"github.com/dgraph-io/dgraph/client"
"github.com/dgraph-io/dgraph/protos/api"
"google.golang.org/grpc"
)
var (
dgraph = flag.String("d", "127.0.0.1:9080", "Dgraph server address")
)
func main() {
flag.Parse()
conn, err := grpc.Dial(*dgraph, grpc.WithInsecure())
if err != nil {
log.Fatal(err)
}
defer conn.Close()
dg := client.NewDgraphClient(api.NewDgraphClient(conn))
resp, err := dg.NewTxn().Query(context.Background(), `blahblah`)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Response: %s\n", resp.Json)
}
Alright, we can improve our import capabilities later on if needed, but for now the DgraphAdapter.mutateFromStream
and gulp db:generate:data
task work well to efficiently import data from our Reddit data dump sources, so we can move onto implementing our front-end Vue components to use that data!
Set Dgraph Schema
Before we add a great deal of data we should take this opportunity to setup a real Dgraph schema so certain predicates are indexed.
Let’s start by creating a new file at
src/dgraph/DgraphSchema.ts
and paste the following into it.export const DgraphSchema = { Comment: ` author: string @index(hash) @count . author_flair_css_class: string @index(hash) . author_flair_text: string @index(hash) . body: string @index(hash, fulltext) . can_gild: bool @index(bool) @count . controversiality: int @index(int) @count . created_utc: dateTime @index(day) . distinguished: string @index(hash) @count . edited: bool @index(bool) @count . gilded: int @index(int) @count . id: string @index(hash) @count . is_submitter: bool @index(bool) @count . link_id: string @index(hash) @count . parent_id: string @index(hash) @count . permalink: string @index(hash) . retrieved_on: dateTime @index(day) . score: int @index(int) @count . stickied: bool @index(bool) @count . subreddit: string @index(hash) . subreddit_id: string @index(hash) . subreddit_type: string @index(hash) . `, Post: ` archived: bool @index(bool) @count . author: string @index(hash) @count . brand_safe: bool . contest_mode: bool @index(bool) @count . created_utc: dateTime @index(day) . domain: string @index(hash) . edited: bool @index(bool) @count . gilded: int @index(int) @count . hidden: bool @index(bool) @count . hide_score: bool @index(bool) @count . id: string @index(hash) . is_crosspostable: bool @index(bool) @count . is_reddit_media_domain: bool @index(bool) . is_self: bool @index(bool) @count . is_video: bool @index(bool) @count . locked: bool @index(bool) @count . no_follow: bool @index(bool) @count . num_comments: int @index(int) . num_crossposts: int @index(int) . over_18: bool @index(bool) @count . parent_whitelist_status: string . permalink: string @index(hash) . pinned: bool @index(bool) . post_hint: string @index(hash) . preview: uid @count @reverse . retrieved_on: dateTime @index(day) . score: int @index(int) . selftext: string @index(hash, fulltext) . send_replies: bool @index(bool) @count . spoiler: bool @index(bool) @count . stickied: bool @index(bool) @count . subreddit: string @index(hash) . subreddit_id: string @index(hash) . subreddit_type: string @index(hash) . thumbnail: string @index(hash) . title: string @index(hash, fulltext) . url: string @index(hash) . whitelist_status: string @index(hash) . ` };
We’ve split the schema into two groups to represent both
Comment
andPost
nodes. It’s worth noting that a handful of the predicates used are shared across both types, but that’s a major advantage to a graph database: We don’t have to explicitly differentiate between each predicate based on what other predicates it is associated with for a given node. For example, thesubreddit
predicate is used in comment and post nodes, but Dgraph will intelligently associate thesubreddit
predicate value with whatever node we’re looking at.Open up the
gulpfile.ts
and delete theconst DGRAPH_SCHEMA
schema declaration at the top and replace it with animport
for the exportedDgraphSchema
constant above.import { DgraphSchema } from './src/dgraph/DgraphSchema';
Also in
gulpfile.ts
we need to adjust thedb:schema:alter
Gulp task to use both schema strings we specified in the other file.gulp.task('db:schema:alter', () => { try { return Promise.all([ new DgraphAdapter().alterSchema(DgraphSchema.Comment), new DgraphAdapter().alterSchema(DgraphSchema.Post) ]); } catch (error) { throw error; } });
Since
DgraphAdapter.alterSchema()
returns aPromise
we can use thePromise.all()
method and pass it a collection of promises. This ensures that thedb:schema:alter
task only completes after the database has been updated using both schema sets.Now drop the existing data and then update the schema with the
db:schema:alter
command.gulp db:drop gulp db:schema:alter
Regenerate Full Data Set
With our schema setup we can now add some significant data to the system that our Vue app can work with. The following commands will add 10,000
submissions and 50,000
comments from the exported Reddit data sets, but feel free to adjust the limit
to suit your needs.
gulp db:generate:data --limit 5000 --path ./src/data/RS_2018-02-01 && gulp db:generate:data --limit 5000 --path ./src/data/RS_2018-02-02
gulp db:generate:data --limit 25000 --path ./src/data/RC_2018-02-01 && gulp db:generate:data --limit 25000 --path ./src/data/RC_2018-02-02
This may take a couple minutes to complete, but once that’s done you’ll have a decently-sized data set to work with for the remainder of the tutorial.
[22:12:16] Requiring external module ts-node/register
[22:12:18] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts
[22:12:18] Starting 'db:generate:data'...
████████████████████████████████████████ 100% | Elapsed: 22s | ETA: 0s | 5000/5000 records
Stream closed, processed 5000 out of 5000 records.
[22:12:40] Finished 'db:generate:data' after 22 s
[22:12:41] Requiring external module ts-node/register
[22:12:42] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts
[22:12:42] Starting 'db:generate:data'...
████████████████████████████████████████ 100% | Elapsed: 24s | ETA: 0s | 5000/5000 records
Stream closed, processed 5000 out of 5000 records.
[22:13:06] Finished 'db:generate:data' after 24 s
[22:13:07] Requiring external module ts-node/register
[22:13:08] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts
[22:13:08] Starting 'db:generate:data'...
████████████████████████████████████████ 100% | Elapsed: 42s | ETA: 0s | 25000/25000 records
Stream closed, processed 25000 out of 25000 records.
[22:13:50] Finished 'db:generate:data' after 42 s
[22:13:51] Requiring external module ts-node/register
[22:13:53] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts
[22:13:53] Starting 'db:generate:data'...
████████████████████████████████████████ 100% | Elapsed: 1m3s | ETA: 0s | 25000/25000 records
Stream closed, processed 25000 out of 25000 records.
[22:14:56] Finished 'db:generate:data' after 1.03 min
Working with Vue
Now that our Dgraph connection is established and we’ve seeded the database with some initial data it’s time to start creating our front-end application pages via Vue. We’ll start by creating the most basic component of a Reddit-like app: the Post list. Below is a rough sketch of what that component should look like when we’ve created it in HTML and CSS.
To help us out we’ll use a Vue framework based on Google’s Material Design specifications called Vuetify. Like other front-end frameworks, it provides some out-of-the-box CSS and custom HTML elements we can use to create our app.
Configuring Vuetify
Add the Vuetify material design framework via
Vue CLI
.vue add vuetify
If you get an error while trying to install the plugin with default settings due to large files in the
src/data
directory try addingsrc/data
to your.gitignore
, then run the Vuetify installer again and select manual configuration using the following settings.? Choose a preset: Configure (advanced) ? Use a pre-made template? (will replace App.vue and HelloWorld.vue) No ? Use custom theme? No ? Use custom properties (CSS variables)? No ? Select icon font Material Icons ? Use fonts as a dependency (for Electron or offline)? No ? Use a-la-carte components? Yes ? Select locale English
After the install completes edit the
src/plugins/vuetify.ts
file that was automatically added and change the lineimport Vuetify from 'vuetify/lib';
toimport Vuetify from 'vuetify';
.import Vue from 'vue'; import Vuetify from 'vuetify'; import 'vuetify/src/stylus/app.styl'; Vue.use(Vuetify, { iconfont: 'md', options: { customProperties: true }, theme: { primary: '#f96315', secondary: '#29b6f6', accent: '#ffc046', info: '#73e8ff', warning: '#c17900', error: '#d32f2f', success: '#43a047' } });
As of the time of writing there is currently a small bug with TypeScript and the Vuetify plugin installation when trying to access the direct
/lib
directory (types declarations cannot be found). The above change fixes that issue.As seen above, we also are overriding the default
theme
property to specify some custom colors, but feel free to play around with those values to get a look you prefer. Theoptions.customProperties
value oftrue
will allow us to explicitly use CSS properties generated by Vuetify throughout the application, so we can reference theme colors and the like within component CSS.You can also opt to use pre-defined colors by adding
import colors from 'vuetify/es5/util/colors';
to the top of the file, then referencing those colors within the customtheme
property.It should be added automatically, but make sure the
src/main.ts
imports this newsrc/plugins/vuetify.ts
file after theVue
import.import Vue from 'vue'; import './plugins/vuetify';
Finally, open the
public/index.html
file and make sure the following stylesheet links exist, which will import the Roboto font and the Material Icons set for us.<!DOCTYPE html> <html lang="en"> <head> <!-- ... --> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:100,300,400,500,700,900" /> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Material+Icons" /> </head> <!-- ... --> </html>
With that, Vuetify should be setup and ready for use as we start creating our own components.
Post Component
Since we’re using TypeScript we gain some benefits of using class-based components within Vue. We can combine the vue-class-component
library with the vue-property-decorator
library to dramatically reduce the lines of code required in our components by using assorted decorators.
.vue
file. The biggest advantage for us is separation of concerns, so each component is self-contained and can be used anywhere we need it throughout the app. Check out the official documentation for far more details on Single File Components.
Start by installing both the
vue-class-component
andvue-property-decorator
libraries.yarn add vue-class-component vue-property-decorator
Create a new
src/components/Post.vue
file and add the following template outline.<template></template> <script lang="ts"> import { Component, Prop, Vue } from 'vue-property-decorator'; @Component export default class Post extends Vue { @Prop() private id!: string; } </script> <style scoped lang="scss"></style>
These three sections are what makeup a
.vue
single file component. The top section contains the HTML template, the middle contains the JavaScript, and the bottom contains the CSS styling. As you can see, by adding thelang="ts"
property to the<script>
element we can tell our editor and parser that we’re using TypeScript. Similarly, thelang="scss"
tells the parser our styling will use Sass.Tip Sass is a mature and powerful CSS extension that works with any normal CSS, but provides a number of useful features such as variables, nesting, and mixins. Check out the full guide for more information on using Sass.The
@Component
decorator that precedes ourPost
class declaration is provided by thevue-class-component
library. It allows us to define parts of our component more succinctly than normal. For example, a normal Vue component would register statedata
by returning an object with child properties.export default { data() { return { id: 1234, name: 'Alice' }; } // ... };
Similarly, computed properties that provide complex logic to otherwise normal component properties are defined within a
computed
object.export default { computed: { lowercaseName() { return this.name.toUpperCase(); } } // ... };
However, with the
@Component
decorator on a Vue class component we can simplifydata
andcomputed
property declarations.@Component export default class extends Vue { id = 1234; name = 'Alice'; get lowercaseName() { return this.name.toUpperCase(); } }
As you can see,
data
properties are now defined by declaring class members andcomputed
properties are class getter methods! Generally, most aspects of Vue component definitions are simplified through the use ofvue-class-decorator
andvue-property-decorator
.
Post Component HTML
Since we’ll be reusing the Post.vue
component we want to define the HTML so it looks like a single row from our Post List mockup image.
We’ll be using Vuetify’s grid system, which is based on the standard CSS flexbox. This allows us to split our layout into a series of responsive columns (12 of them in this case).
We’ll start by adding the
<v-layout>
element within the root<template>
element.<template> <v-layout class="post" row wrap my-2> </v-layout> </template>
The grid system uses a progressive series of elements:
<v-container>
- The base element of a grid system. Should contain one or more<v-layout>
elements.<v-layout>
- Similar to a<v-container>
, but multiple<v-layout>
can exist within a single<v-container>
, providing the ability for grids within other grids.<v-flex>
- The “content holder” element of a grid. The underlyingflex
CSS property of a<v-flex>
is set to1
, which means a series of<v-flex>
elements will attempt to responsively fill out the space they are given within their parent<v-layout>
.
We’ll actually specify the parent
<v-container>
element in another component further up the chain, since we don’t want each individualPost
element to be a container unto itself. The<v-layout>
row
class ensures we’re flexing across rows (not columns). We also want children to be able to wrap if needed. Finally, we’ll make heavy use ofmargin
and/orpadding
throughout this component by using the Vuetify CSS spacing classes. So, themy-2
class translates intomargin: 2px 0;
since we want a 2-pixel margin along the y-axis (top and bottom).Our
Post
layout has three distinct horizontal sections: Voting, thumbnail image, and post content. Therefore, we’ll split each of those sections into their own<v-flex>
element. Let’s start by adding the voting flexbox.<template> <v-layout class="post" row wrap my-2> <v-flex class="votes" xs1 px-1 mx-1> <v-icon class="arrow up accentuated">arrow_upward</v-icon> <span class="score">1234</span> <v-icon class="arrow down accentuated">arrow_downward</v-icon> </v-flex> </v-layout> </template>
We’re making use of the Material Icons pack that is part of the library, in addition to adding some helper classes we’ll use later.
Info Thexs1
CSS class helper seen in the<v-flex>
element above works like many other responsive frameworks.xs
is one of Vuetify’s display options and it sets a breakpoint for viewports under600px
. We don’t really want to worry about viewports for this tutorial, so using thexs
extreme lets us effectively not have a breakpoint to worry about (since all displays should meet that criteria). The1
followingxs
is the number of columns our flexbox is spanning.Next, let’s add another flexbox as a sibling to the
.votes
flexbox with the.thumbnail
class.<template> <v-layout class="post" row wrap my-2> <v-flex class="votes" xs1 px-1 mx-1> <v-icon class="arrow up accentuated">arrow_upward</v-icon> <span class="score">1234</span> <v-icon class="arrow down accentuated">arrow_downward</v-icon> </v-flex> <v-flex class="thumbnail" xs1 px-1 mx-1> <a href="#"> <v-img :src="`https://lorempixel.com/70/70`" :lazy-src="`https://dummyimage.com/70x70/f5f5f5/f96515&text=D`" aspect-ratio="1" height="70" width="70" /> </a> </v-flex> </v-layout> </template>
We’re pulling some placeholder images for now and setting their size to
70x70
pixels. The special:lazy-src
<v-img>
property is helpful when you want to display a temporary image while the full, normal image loads in the background.Our third flexbox is the
.content
class.<template> <v-layout class="post" row wrap my-2> <v-flex class="votes" xs1 px-1 mx-1> <v-icon class="arrow up accentuated">arrow_upward</v-icon> <span class="score">1234</span> <v-icon class="arrow down accentuated">arrow_downward</v-icon> </v-flex> <v-flex class="thumbnail" xs1 px-1 mx-1> <a href="#"> <v-img :src="`https://lorempixel.com/70/70`" :lazy-src="`https://dummyimage.com/70x70/f5f5f5/f96515&text=D`" aspect-ratio="1" height="70" width="70" /> </a> </v-flex> <v-flex class="content" xs10 px-1 mx-1> <span class="title"> <a href="#" class="text--primary" >Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent eu maximus sem. Aliquam erat volutpat. Aliquam maximus efficitur ligula eu vestibulum.</a > <span class="domain text--secondary caption ml-1 font-weight-bold" >(<a href="/r/AskReddit" class="text--secondary accentuated" >self.AskReddit</a >)</span > </span> <span class="tagline caption"> submitted 5 hours ago by <a class="accentuated" href="/user/JustSomeGuy">JustSomeGuy</a> to <a class="accentuated" href="/r/AskReddit">r/AskReddit</a> </span> <ul class="buttons font-weight-medium"> <li class="comment"> <a href="#" class="text--secondary accentuated">75 comments</a> </li> <li class="share"> <a href="#" class="text--secondary accentuated">share</a> </li> <li class="save"> <a href="#" class="text--secondary accentuated">save</a> </li> <li class="toggle"> <a href="#" class="text--secondary accentuated">hide</a> </li> <li class="award"> <a href="#" class="text--secondary accentuated">give award</a> </li> <li class="report"> <a href="#" class="text--secondary accentuated">report</a> </li> <li class="crosspost"> <a href="#" class="text--secondary accentuated">crosspost</a> </li> </ul> </v-flex> </v-layout> </template>
We want the content to take up the majority of the remaining space, so we set the flexbox width to
xs10
to take up 10 out of the 12 total columns. Thetext--primary
andtext--secondary
classes are references to theme CSS properties. It’s helpful to use such classes wherever possible so that we can change the look of the entire app with just a few color changes to the theme.We’re also playing with the font weighting for a number of elements, just to make things appear more like they do in the actual Reddit.
Post Component CSS
Alright, we’ve got our rough HTML layout but we need to add some additional custom styling beyond the helper classes we used from Vuetify.
Update the <style></style>
section of src/components/Post.vue
to look like the following.
<style scoped lang="scss">
.post {
.votes {
max-width: 40px;
text-align: center;
* {
display: block;
}
}
.thumbnail {
max-width: 70px;
}
.content {
a {
text-decoration: none;
}
.buttons {
display: block;
list-style-type: none;
padding: 1px 0;
li {
display: inline-block;
line-height: 1.5em;
padding-right: 0.33em;
}
}
.title {
display: block;
font-weight: bold;
}
}
}
</style>
We won’t go into much detail here since most of this is basic CSS, but it’s worth mentioning that the use of Sass lets us nest our CSS selectors. This means that rules inside .post { .votes { ... } }
will only apply to .votes
found within .post
, but not elsewhere. It also helps to visually mimic the hierarchical structure found in the HTML.
Post Component Script
At this point, we aren’t actually implementing any logic into the Post
component, so the script section can be left as is. We’ll come back to it shortly once we have our layout looking like we want. For now, let’s move up the chain and create the PostList
component that will use instances of our Post
component.
PostList Component
Create a new src/components/PostList.vue
file and add the following to it.
<template>
<v-container grid-list-xs>
<Post v-for="i of 20" :key="i"></Post>
</v-container>
</template>
<script lang="ts">
import { Component, Prop, Vue } from 'vue-property-decorator';
import Post from '@/components/Post.vue';
@Component({
components: { Post }
})
export default class PostList extends Vue {}
</script>
<style scoped lang="scss"></style>
As mentioned before, here is where we’ve added the <v-container>
element that specifies we want a grid list. Within that container we’re using a v-for
loop to render a list of Post
elements. Normally v-for
would be used to render a collection of objects from data, but for testing purposes we’re just using a collection of numbers. As with React
and other frameworks, we must ensure we pass a unique :key
property value when iterating through a list.
The script is fairly simple and similar to what we saw in the Post
component, but we’re passing an object to the @Component
directive and specifying that our components
property contains the Post
component. Now all we have to do is get our app to render the PostList
component and we’ll be in business!
Handling Global Sass Variables
We need a convenient way to inject custom CSS throughout the app, which we can do by modifying the CSS Webpack loader.
Create a
src/assets/css/main.scss
file and add the following Sass..accentuated { &:hover { color: var(--v-accent-base) !important; } }
Notice that we’re using the
--v-accent-base
variable, which is generated automatically because we passedoptions.customProperties
to the Vuetify declaration in Configuring Vuetify.We’ll use this file for global CSS that we need access to throughout the app. However, in order for
scoped
CSS within a.vue
file to have access to the globalsrc/assets/css/main.scss
file we need to modify the Webpack loadercss
settings so it’ll import the file automatically.Open
vue.config.js
and add the followingcss: { ... }
property.module.exports = { // ... css: { loaderOptions: { sass: { data: `@import "~@/assets/css/main.scss";` } } } };
Now in the
src/components/Post.vue
component where we use theaccentuated
class we’ll see:hover
effects using the accent base color specified in our Vuetify theme.Warning Unfortunately, Vuetify defaults to using the!important
flag for a number of its generated CSS classes. Therefore, if you notice custom CSS changes aren’t taking affect you may have to resort to adding the!important
flag to your own CSS rules intended to take precedence.
Updating the App and Home Components
The default Vue layout looks neat and all, but we obviously need to get rid of that starter stuff so our app functions like we want.
Delete the
src/views/About.vue
file.rm src/views/About.vue
Open
src/App.vue
and change the contents to the following.<template> <v-app> <v-navigation-drawer app> <router-link to="/">Home</router-link> </v-navigation-drawer> <v-toolbar app></v-toolbar> <v-content> <v-container fluid> <router-view></router-view> </v-container> </v-content> <v-footer app></v-footer> </v-app> </template> <style lang="scss"></style>
This gives us a basic app layout with a navigation bar, a footer, a toolbar, and our primary content section.
Open
src/views/Home.vue
and modify the contents to the following.<template> <PostList /> </template> <script lang="ts"> import { Component, Vue } from 'vue-property-decorator'; import PostList from '@/components/PostList.vue'; @Component({ components: { PostList } }) export default class Home extends Vue {} </script> <style lang="scss"></style>
This should look familiar as the structure is almost identical to what we saw in the
PostList
component.That’s it! Now just run the dev server with
yarn serve
to launch the updated app.yarn serve
It should look something like the above screenshot.
Querying Dgraph
Now that our PostList
component is configured it’s time to populate it with actual data from Dgraph.
Managing State With the Vuex Library
As is common practice with Vue applications we’ll be using the vuex
library which provides common statement management patterns similar to those found in React/Redux and the like. Similar to Redux, Vuex uses a combination of actions and mutations to perform one-way transactions. An action never modifies the state and, instead, merely provides instruction for a mutation to perform actual state changes.
Vuex will already be installed if you used the same configuration found in Create a Vue CLI Project. However, if not, feel free to add it manually via npm or yarn.
yarn add vuex
We’re going to store everything about our state management in the
src/state
directory, so create that now if needed.mkdir state
Add the following
import
to yoursrc/main.ts
file.import { store } from '@/state/store';
This file will be the entry point for all Vuex store management.
Create the following files within the
src/state
directory.actions.ts
index.ts
mutations.ts
state.ts
store.ts
types.ts
Open
src/state/store.ts
and add the following code to it.import Vuex from 'vuex'; import Vue from 'vue'; import { Actions } from '@/state/actions'; import { Mutations } from '@/state/mutations'; import { State } from '@/state/state'; Vue.use(Vuex); export const store = new Vuex.Store({ actions: Actions, mutations: Mutations, state: State });
This file instantiates a new
Vuex.Store
instance and sets the three critical properties to exported values that we’ll define in a moment.Open the
src/state/types.ts
file and paste the following into it.export const Types = { Action: { Post: { Get: { Paginated: 'Post.Get.Paginated' } } }, Mutation: { Post: { Set: { Paginated: 'Post.Set.Paginated' } } } };
Vuex expects a given action or mutation to be defined by a unique
string
value key. However, it is useful to use enumerations or other static options to define these keys so you don’t need to manually remember and enter the names of the actions or mutations you’re performing. That’s what thetypes.ts
file above accomplishes. It will allow us to reference potentially complex action or mutation names through values the editor will verify.Open
src/state/actions.ts
and add the following code.import { Types } from '@/state/types'; import { DgraphAdapter } from '@/dgraph/DgraphAdapter'; export const Actions = { async [Types.Action.Post.Get.Paginated]( { commit }: { commit: any }, { first = 50, offset = 0 }: { first?: number; offset?: number } ) { const { data } = await new DgraphAdapter().query( `query posts($first: int, $offset: int) { data(func: has(domain), first: $first, offset: $offset) @filter((not has(crosspost_parent)) and eq(over_18, false)) { uid expand(_all_) { uid expand(_all_) } } }`, { $first: first, $offset: offset } ); commit(Types.Mutation.Post.Set.Paginated, { posts: data }); } };
Here is where we’re defining the actual actions that we want to be able to dispatch. The
Actions
object is just a collection of functions and we’re naming them using the pre-definedTypes
found in thesrc/state/types.ts
file. In this case, we’ve defined anasync
action named the value ofTypes.Action.Post.Get.Paginated
.Vuex actions pass a
context
parameter which allows us to access the state (context.state
) or commit a mutation (context.commit
). Here, we only need access to thecommit
function, so we’re destructuring it in the parameter definition (which is a common pattern when using Vuex). We’re also accepting a second set of custom arguments which we’ll use to adjust the logic of the action. Remember, actions can be asynchronous but cannot modify the state, but should just commit a mutation informing the state of a potential change.This GraphQL+- query we’re performing defines two GraphQL Variables (
$first
and$offset
), which allows us to pass arguments to Dgraph to dynamically modify the query. Bothfirst
andoffset
filters are part of the built-in pagination options. Thus, the default values of50
and0
, respectively, will return the first50
posts. The extra@filter
directives used here are just to narrow the search down so we don’t get any crossposts, nor anything that might be NSFW.If you have Dgraph running locally you can test that query below.
query posts($first: int, $offset: int) { data(func: has(domain), first: 50, offset: 0) @filter((not has(crosspost_parent)) and eq(over_18, false)) { uid expand(_all_) { uid expand(_all_) } } }
curl http://127.0.0.1:8080/query -XPOST -d '
query posts($first: int, $offset: int) { data(func: has(domain), first: 50, offset: 0) @filter((not has(crosspost_parent)) and eq(over_18, false)) { uid expand(_all_) { uid expand(_all_) } } } ' | python -m json.tool | lesspackage main import ( "context" "flag" "fmt" "log" "github.com/dgraph-io/dgraph/client" "github.com/dgraph-io/dgraph/protos/api" "google.golang.org/grpc" ) var ( dgraph = flag.String("d", "127.0.0.1:9080", "Dgraph server address") ) func main() { flag.Parse() conn, err := grpc.Dial(*dgraph, grpc.WithInsecure()) if err != nil { log.Fatal(err) } defer conn.Close() dg := client.NewDgraphClient(api.NewDgraphClient(conn)) resp, err := dg.NewTxn().Query(context.Background(), `blahblah`) if err != nil { log.Fatal(err) } fmt.Printf("Response: %s\n", resp.Json) }
As you may recall from Querying Dgraph the call to
DgraphAdapter().query()
lets us pass optional arguments, and if they exist it will invoke thetxn.queryWithVars()
method from thedgraph-js-http
library.Once the result of our query has returned we finish by calling the
commit()
method to invoke the appropriate mutation. Since the action name wasPost.Get.Paginated
to invoke a retrieval of posts that we pass as the payload argument to our mutation, the mutation we’ll commit isPost.Set.Paginated
. We could name these anything we want and may want to change them in the future, but this seems like an appropriate name for a mutation that changes the paginated post list.Speaking of mutations, open
src/state/mutations.ts
and paste the following into it.import { Types } from '@/state/types'; export const Mutations = { [Types.Mutation.Post.Set.Paginated](state: any, { posts }: { posts: any[] }) { state.posts = [...state.posts, ...posts]; } };
As with the
Actions
object exported fromsrc/state/actions.ts
, theMutations
object is a collection of mutation methods. The first parameter provided by Vuex is the current state, which is required and will be used to update or mutate the state within the handler function. We’ve also opted to pass an optional second argument that contains custom data used to process this mutation. Here we’re destructuring theposts
property that was passed via thecommit()
method in our action, and setting thestate.posts
value to it.The final step is to open
src/state/state.ts
and set the initial state values for any state properties we’ll be using. In this case, we just have theposts
property used above.export const State = { posts: [] };
Using State in the PostList Component
Now that our state is configured and we can extract some paginated post data we need to add that functionality to our src/components/PostList.vue
component.
Open
src/components/PostList.vue
and add the followinggetPosts()
computed property andcreated()
lifecycle method to thePostList
class. Don’t forget the new{ Types }
import
as well.<script lang="ts"> import { Component, Prop, Vue } from 'vue-property-decorator'; import Post from '@/components/Post.vue'; import { Types } from '@/state'; @Component({ components: { Post } }) export default class PostList extends Vue { get getPosts() { return this.$store.state.posts; } public async created() { // Get post list. await this.$store.dispatch(Types.Action.Post.Get.Paginated, { first: 100, offset: 0 }); } } </script>
Vue has a number of component lifecycle hooks, one of which is
created
. Thevue-class-component
library lets us add run code during these lifecycle hooks by declaring class methods with the matching names and passing functions that should be executed during those hooks. Thus, thepublic async create()
method fires after thePostList
component instance is created. In it we await the result dispatching thePost.Get.Paginated
action with extra optional arguments. As we saw above, this will retrieve the data from Dgraph and then commit a mutation to update the state.The
getPosts
getter is a computed property, which means that Vue will intelligently evaluate the value of this property and dynamically re-render any components that rely on this property when the value changes. Therefore, when thestate.posts
property changes, the value of thegetPosts
property is also updated.To make use of
getPosts
let’s update thePostList.vue
HTML section as seen below.<template> <v-container grid-list-xs> <Post v-for="post in getPosts" :key="post.id" v-bind="post"></Post> </v-container> </template>
Vue provides a number of helper directives which are HTML attributes that begin with
v-
.v-for
- Loops over a collection. We used this before to loop over a collection of numbers for dummy data, but here we’re using it to iterate over the collection returned by thegetPosts
computed property seen above.v-bind
- Dynamically binds an attribute to a value. Typically this is written in the form ofv-bind:attr-name="value"
, but if we exclude the attribute name then Vue will automatically pass (i.e.bind
) every property of the object in question to the component.:key
-v-bind
also has a shorthand syntax that lets us avoid typing thev-bind
prefix. By using just the colon followed by the attribute we can replicate a binding, so here we’re binding thekey
attribute to the value ofpost.id
.
The PostList
component is updated and is properly passing data to the Post
instances it creates, but we need to update the Post
component to actually display that data.
Binding Data in the Post Component
Open
src/components/Post.vue
and change the<script>
section to the following.<script lang="ts"> import { Component, Prop, Vue } from 'vue-property-decorator'; @Component export default class Post extends Vue { @Prop(String) private id!: string; @Prop(String) private author!: string; @Prop(String) private created_utc!: Date; @Prop(String) private domain!: string; @Prop(Boolean) private is_self!: boolean; @Prop(Number) private num_comments!: number; @Prop(String) private permalink!: string; @Prop(Number) private score!: number; @Prop(String) private subreddit!: string; @Prop(String) private thumbnail!: string; @Prop({ default: 70 }) private thumbnail_height!: number; @Prop({ default: 70 }) private thumbnail_width!: number; @Prop(String) private title!: string; @Prop(String) private url!: string; get authorUrl() { return `/user/${this.author}`; } get domainUrl() { if (this.is_self) { return this.subredditUrl; } else { return `/domain/${this.domain}`; } } get fullUrl() { return this.is_self ? this.permalink : this.url; } get hasAuthor() { return this.author !== '[deleted]'; } get subredditUrl() { return `/r/${this.subreddit}`; } get thumbnailUrl() { if (this.thumbnail === 'self') { return require('../assets/images/thumbnail-self.png'); } else if (this.thumbnail === 'default') { return require('../assets/images/thumbnail-default.png'); } else { return this.thumbnail; } } } </script>
This may look a bit overwhelming at first, but really we’ve just added two types of data to the
Post
component: properties and computed properties. Let’s start with the properties list. These properties are defined using the@Prop
decorator to specify their types, name, default values, and so forth. The names are taken directly from the properties of our Dgraph predicates used by a Post node.As mentioned before, computed properties are specified by a getter method within the class component, so we’re using such computed properties to “calculate” additional logic. We won’t go over them all, but
thumbnailUrl()
is a good example as it allows us to return the proper post thumbnail URL based on the possible values found in the database.Tip ThethumbnailUrl()
property references two custom thumbnail images which you can download and add to yoursrc/assets/images/
directory to include them in your own project. They can be found in the src/assets/images directory of the GitHub repository.Next, let’s update the HTML section of the
Post.vue
component. We’ll go through each of the threev-flex
elements one at time.<v-flex class="votes" xs1 px-1 mx-1> <v-icon class="arrow up accentuated">arrow_upward</v-icon> <span class="score">{{ score }}</span> <v-icon class="arrow down accentuated">arrow_downward</v-icon> </v-flex>
The only change here is to use the actual
score
property passed to thePost
component instance. Vue’s text interpolation syntax merely requires surrounding a property value with double curly braces (aka “mustaches”). This syntax is used all the time within Vue templates, so you’ll see it frequently.The second flexbox section should be updated as seen below.
<v-flex class="thumbnail" xs1 px-1 mx-1> <a :href="fullUrl"> <v-img :src="thumbnailUrl" :lazy-src="thumbnailUrl" aspect-ratio="1" height="70" width="70" /> </a> </v-flex>
We’re no longer using static URL strings, but instead are binding
:href
,:src
, and:lazy-src
attributes to computed property functions.The last flexbox should look like the following.
<v-flex class="content" xs10 px-1 mx-1> <span class="title"> <a :href="fullUrl" class="text--primary">{{ title }}</a> <span class="domain text--secondary caption ml-1 font-weight-bold" >(<a :href="domainUrl" class="text--secondary accentuated" >{{ domain }}</a >)</span > </span> <span class="tagline caption"> submitted {{ created_utc | moment('from') }} by <a class="accentuated" :href="authorUrl" v-if="hasAuthor" >{{ author }}</a > <span v-else>{{ author }}</span> to <a class="accentuated" :href="subredditUrl">r/{{ subreddit }}</a> </span> <ul class="buttons font-weight-medium"> <li class="comment"> <a :href="permalink" class="text--secondary accentuated" >{{ num_comments }} comments</a > </li> <li class="share"> <a href="#" class="text--secondary accentuated">share</a> </li> <li class="save"> <a href="#" class="text--secondary accentuated">save</a> </li> <li class="toggle"> <a href="#" class="text--secondary accentuated">hide</a> </li> <li class="award"> <a href="#" class="text--secondary accentuated">give award</a> </li> <li class="report"> <a href="#" class="text--secondary accentuated">report</a> </li> <li class="crosspost"> <a href="#" class="text--secondary accentuated">crosspost</a> </li> </ul> </v-flex>
Quite a lot has changed here, but the same rules as techniques used in the previous sections apply. The first notable addition is
submitted {{ created_utc | moment('from') }}
. The pipe character indicates a Vue filter function, which simplifies text formatting within Vue templates. In this case, we’re using a specialmoment()
filter function to transform thecreated_utc
date into a human-readable “X seconds ago” format.The other unknown addition is just below that in which we use the
v-if
andv-else
directives to determine if the post has a valid author. If so, a<a>
link element is added to link to the author URL. Otherwise, no link is set and the author is printed in plain text. This mimics the Reddit behavior of[deleted]
users who no longer have a user page, but may still have active comments or posts.The last step is to add the
vue-moment
library, which provides a filter function we can use in Vue that behaves similar toMoment.js
.yarn add vue-moment
Open the
src/main.ts
file and add the following code to import and usevue-moment
.import VueMoment from 'vue-moment'; Vue.use(VueMoment);
Alright, our Post
component is updated and ready to display the data it receives from the parent PostList
component. Save everything and run yarn serve
to check out the new post list, which will show the first 100
post records in the database.
Adding Infinite Scrolling
The post list is working well, but one it’s missing one of the defining features of mass-content sites like Reddit: infinite scrolling. When we scroll to the bottom of the post list it should immediately load the next page of posts inline, without loading a new page.
Let’s start by adding the vue-mugen-scroll library that includes some built-in helper functionality. For example, we could add infinite scrolling pretty easily by checking the current scroll vs the window height, but we’d also need to perform a number of sanity checks like debouncing. Using
vue-mugen-scroll
solves this for us.yarn add vue-mugen-scroll
Open
src/components/PostList.vue
and the following<mugen-scroll>
element immediately after our<Post>
element.<template> <v-container grid-list-xs> <Post v-for="post in getPosts" :key="post.id" v-bind="post"></Post> <mugen-scroll :handler="getPaginatedPosts" :handleOnMount="false"> Loading... </mugen-scroll> </v-container> </template>
This tells the scroller to invoke the
getPaginatedPosts()
method when the bottom of the viewport is reached. Since our component loads the first data set on its own we don’t need to invoke the handler on mount.In the
<script>
sectionimport MugenScroll
and add the newgetPaginatedPosts()
method.// @ts-ignore import MugenScroll from 'vue-mugen-scroll'; @Component({ components: { MugenScroll, Post } }) export default class PostList extends Vue { get getPosts() { return this.$store.state.posts; } public async getPaginatedPosts() { // Get post list. await this.$store.dispatch(Types.Action.Post.Get.Paginated, { first: 100, offset: this.$store.state.posts.length }); } public async created() { // Get post list. await this.getPaginatedPosts(); } }
Since we’re using the
mugen-scroll
component in the HTML section we need to include it in the referencedcomponents
property. We’ve also adjusted the logic of the component slightly, so thecreated()
method just calls thegetPaginatedPosts()
method directly, so all our retrieval logic is in one spot. Retrieving posts is nearly identical to how it worked before, but we’ve changed the passedoffset
argument to equal to current length of thestate.posts
property. This handles pagination of the dataset automatically.
That’s all there is to it! If we save our changes and run yarn serve
to see the new post list we can now infinitely scroll as seen below.
Adding Link Views
Now that our front page post list is complete the next milestone is to allow us to view the content of a given post. Although it may be a bit confusing at first, Reddit’s site terminology refers to the actual content of a given submission as a link. This makes sense when a submission is a outside URL to another resource, but the same term is also used to describe self
submissions that remain within Reddit.
Therefore, while the front page post list view already handles outside content links for us, we need a component and router rule to handle the inner comment thread of a given link, which is in the form of /r/:subreddit/comments/:link/:slug
.
Creating Custom Routes
Routing is provided by vue-router
and can be configured quite easily. We already have the default route of /
that renders the Home
component, so we need to add another route to handle the /r/:subreddit/comments/:link/:slug
route.
- Open the
src/router.ts
file. Add a new entry into the
routes
property with apath
of'/r/:subreddit/comments/:link/*'
.import Link from '@/components/Link.vue';/ // .. export default new Router({ mode: 'history', base: process.env.BASE_URL, routes: [ { path: '/', name: 'home', component: Home }, { path: '/r/:subreddit/comments/:link/*', name: 'link', component: Link } ] });
Make sure to also import the
Link
component at the top. This route will now match link-specific URLs.
Adding Comments to the State
Since an internal link view displays a list of discussion comments we need to add comments to the app state and appropriate actions and mutations to update that property.
Open
src/state/state.ts
and add thecomments
property with a default value of an empty array.export const State = { comments: [], posts: [] };
Open
src/state/types.ts
and add the newComment.Get.Paginated
action andComment.Set.Paginated
mutation types.export const Types = { Action: { Comment: { Get: { Paginated: 'Comment.Get.Paginated' } }, Post: { Get: { Paginated: 'Post.Get.Paginated' } } }, Mutation: { Comment: { Set: { Paginated: 'Comment.Set.Paginated' } }, Post: { Set: { Paginated: 'Post.Set.Paginated' } } } };
Open
src/state/mutations.ts
and add a new entry for theComment.Set.Paginated
mutation method.import { Types } from '@/state/types'; export const Mutations = { [Types.Mutation.Comment.Set.Paginated]( state: any, { comments }: { comments: any[] } ) { state.comments = [...state.comments, ...comments]; }, // ... };
Similar to how we update
posts
,state.comments
are combined with existing comments in the state since this pagination mutation incrementally adds data.Next, open
src/state/actions.ts
and add the following entry for theComment.Get.Paginated
action method.export const Actions = { async [Types.Action.Comment.Get.Paginated]( { commit }: { commit: any }, { link, first = 50, offset = 0 }: { link: string; first?: number; offset?: number } ) { const { data } = await new DgraphAdapter().query( `query comments($link: string, $first: int, $offset: int) { data(func: eq(link_id, $link), first: $first, offset: $offset) { uid expand(_all_) { uid expand(_all_) } } }`, { $link: `t3_${link}`, $first: first, $offset: offset } ); commit(Types.Mutation.Comment.Set.Paginated, { comments: data }); }, // ... }
This action invokes a GraphQL+- query to get all comments based on the passed
link
id argument. You’ll recall the route we’ll be parsing of'/r/:subreddit/comments/:link/*'
. The:link
param from that URL is in the database under thelink_id
predicate. The values are unique strings that look something like this:t3_7ub10c
.Reddit IDs are a unique Base36 value prefixed by the relevant Reddit content type as seen below.
Prefix Type t1_ Comment t2_ Account t3_ Link t4_ Message t5_ Subreddit t6_ Award Therefore, a comment with a
link_id
predicate value oft3_7ub10c
corresponds to a submission/post with the id7ub10c
. That’s why our query parameter object above prefixes thelink
value witht3_
.
Creating the Comment Component
Now that we can detect link routes and our state is configured to retrieve comments based on those matching link IDs we can add a link view. Reddit link views are just a collection of comments that makeup that link’s discussion, so we’ll start with the Comment.vue
component.
- Create a new
src/components/Comment.vue
file. Add the following
<template>
HTML.<template> <v-layout class="comment" row wrap my-2> <v-flex class="votes" xs1 px-1 mx-1> <v-icon class="arrow up accentuated">arrow_upward</v-icon> <v-icon class="arrow down accentuated">arrow_downward</v-icon> </v-flex> <v-flex class="content" xs11 px-1 mx-1> <span class="tagline caption"> <a class="accentuated" :href="authorUrl" v-if="hasAuthor">{{ author }}</a> <span v-else>{{ author }}</span> <span class="score">{{ score }} points</span> {{ created_utc | moment('from') }} </span> <span class="body" v-html="body"></span> <ul class="buttons font-weight-medium"> <li class="permalink"> <a :href="permalink" class="text--secondary accentuated">permalink</a> </li> <li class="source"> <a href="#" class="text--secondary accentuated">share</a> </li> <li class="embed"> <a href="#" class="text--secondary accentuated">save</a> </li> <li class="save"> <a href="#" class="text--secondary accentuated">hide</a> </li> <li class="report"> <a href="#" class="text--secondary accentuated">report</a> </li> <li class="award"> <a href="#" class="text--secondary accentuated">give award</a> </li> <li class="reply"> <a href="#" class="text--secondary accentuated">reply</a> </li> <li class="hide-children"> <a href="#" class="text--secondary accentuated" >hide child comments</a > </li> </ul> </v-flex> </v-layout> </template>
Reddit uses very similar formatting for comments as it does for posts, so we’re able to reuse a significant amount of the layout found in the
Post.vue
. A comment consists of voting arrows on the side, the author at the top, the score of that comment, created date, body text of the actual comment, and then a series of interactive buttons at the bottom.There are a lot of properties we’re using within the template, so the majority of the
<script>
section is just property definition.<script lang="ts"> import { Component, Prop, Vue } from 'vue-property-decorator'; @Component export default class Comment extends Vue { @Prop(String) private id!: string; @Prop(String) private author!: string; @Prop(String) private author_flair_text!: string; @Prop(String) private body!: string; @Prop(Boolean) private can_gild!: boolean; @Prop(Number) private controversiality!: number; @Prop(String) private created_utc!: Date; @Prop(String) private distinguished!: string; @Prop(Boolean) private edited!: boolean; @Prop(Number) private gilded!: number; @Prop(Boolean) private is_submitter!: boolean; @Prop(String) private link_id!: string; @Prop(String) private parent_id!: string; @Prop(String) private permalink!: string; @Prop(String) private retrieved_on!: Date; @Prop(Number) private score!: number; @Prop(Boolean) private stickied!: boolean; @Prop(String) private subreddit!: string; @Prop(String) private subreddit_id!: string; @Prop(String) private subreddit_type!: string; get authorUrl() { return `/user/${this.author}`; } get hasAuthor() { return this.author !== '[deleted]'; } get subredditUrl() { return `/r/${this.subreddit}`; } } </script>
Lastly, add the following
<style>
section.```css ```
The Comment.vue
component is now ready to go, we just have to pass in a comment prop it can display.
Creating the Link Component
A Link.vue
component will behave similar to the PostList.vue
component by acting as a parent view for comments.
- Create a new Vue file at
src/components/Link.vue
. Add the following to the HTML template section.
<template> <v-container grid-list-xs> <Comment v-for="comment in getComments" :key="comment.id" v-bind="comment" ></Comment> <mugen-scroll :handler="getPaginatedComments" :handleOnMount="false" ></mugen-scroll> </v-container> </template>
Here we’re just creating a container and looping through all
Comments
provided by thegetComments
method. We’ll also use the same infinite scrolling helper we used in thePostList.vue
.Let’s next add the
<script>
section with the following code.<script lang="ts"> import { Component, Prop, Vue } from 'vue-property-decorator'; import Comment from '@/components/Comment.vue'; import { Types } from '@/state'; // @ts-ignore import MugenScroll from 'vue-mugen-scroll'; @Component({ components: { MugenScroll, Comment } }) export default class Link extends Vue { private loading = false; get getComments() { return this.$store.state.comments; } public async created() { await this.getPaginatedComments(); } public async getPaginatedComments() { await this.$store.dispatch(Types.Action.Comment.Get.Paginated, { link: this.$route.params.link, first: 50, offset: this.$store.state.comments.length }); } } </script>
The logic is similar to what we used in
PostList.vue
. thegetPaginatedComments()
method dispatches theComment.Get.Paginated
action and retrieves the first50
comments with an offset of the currentstate.comments
length. We also pass the currentlink
param obtained from the route. As we configured in the router, a route in the form of/r/:subreddit/comments/:link/*
passes the:link
param ID to the action and performs a Dgraph query to retrieve matching records.
Optimizing the Imported Data Set
We’re all set and should now be able to navigate to any link URL (i.e. /r/:subreddit/comments/:link/*
). When viewing the home page post list these links are found in the post content for self.
posts or by clicking the comments button of a post.
However, while this functionality works, we’re running into an unforeseen problem: A number of link views simply show no comments.
The problem here is our imported data set is incomplete. While we have tens of thousands of comments in the database, Reddit receives millions of comments per day, so our data set that contains data from just two days of submissions and comments contains a lot of “gaps.” It isn’t all that likely that a given submission within those two days also has a comment from those two days and is one of the comments we imported.
The solution is to add some minor logic to our data import functions and perform a new import. We’ll implement a few simple rules to dramatically improve the quality of our imported Reddit data, without having to add massive amounts of extra data or download more data dump files.
- Import comments into the database first.
- Import submission posts second, but run each of them through a validation function that checks the following criteria:
- The post must have at least one comment. While this isn’t accurate to the real Reddit, it’s better for our purposes in this tutorial to only look at posts that were commented on.
- At least one comment made on the post should be within the collection of comments already contained in the database.
With these validation rules setup we can ensure that we’re only adding posts that contain comments that are within the Dgraph database, which will eliminate the empty link view seen above.
Adjusting the Dgraph Mutation from Stream Logic
The DgraphAdapter.mutationFromStream
method works great for importing from our massive data sets, but we need to add some additional logic so we can validate data before pushing it to the database.
Open src/dgraph/DgraphAdapter.ts
and update the mutationFromStream()
method to look like the following.
public static async mutateFromStream({
stream,
batchSize = 50,
limit = 150,
offset = 0,
validator
}: {
stream: ReadStream;
batchSize?: number;
limit?: number;
offset?: number;
validator?: (data: any) => void;
}) {
const adapter = new DgraphAdapter();
let batch: any[] = [];
let invalidCount = 0;
let skippedCount = 0;
let totalCount = 0;
const bar = new CliProgress.Bar(
{
stopOnComplete: true,
format:
'{bar} {percentage}% | Elapsed: {duration_formatted} | ETA: {eta_formatted} | {value}/{total} records'
},
CliProgress.Presets.shades_classic
);
// Start progress bar with maximum of limit.
bar.start(limit, 0);
const syncMutation = async (readStream: ReadStream, event?: string) => {
try {
// Pause during async.
readStream.pause();
// Mutate batch
const response = await adapter.mutate({ request: batch });
// Reset batch.
batch = [];
// Update progress bar.
bar.update(totalCount);
// Resume after async.
readStream.resume();
} catch (error) {
// Stop progress bar.
bar.stop();
console.log(error);
}
};
return new Promise((resolve, reject) => {
stream
.pipe(es.split())
.pipe(es.parse())
.on('data', async function(this: ReadStream, data: any) {
if (offset && offset > 0 && skippedCount < offset) {
skippedCount++;
} else if (validator ? validator(data) : true) {
// Add data to batch and update total count.
batch.push(data);
totalCount++;
if (totalCount >= limit) {
// Close stream if total exceeds limit.
this.destroy();
} else if (batch.length === batchSize) {
// Synchronously mutate if batch length meets batchSize.
await syncMutation(this, 'data');
}
} else {
invalidCount++;
}
})
.on('error', (error: Error) => {
// Stop progress bar.
bar.stop();
console.log(error);
reject(error);
})
.on('close', async function(this: ReadStream) {
// Synchronously mutate if batch contains any extraneous records.
if (batch.length > 0) {
await syncMutation(this, 'close');
}
// Stop progress bar.
bar.stop();
resolve(
`Processed ${totalCount +
invalidCount +
skippedCount} records (${totalCount} mutated, ${invalidCount} invalid, ${skippedCount} skipped).`
);
});
});
}
We added two new parameters: offset?: number;
and validator?: (data: any) => void;
. We won’t need to use offset
here, but it seemed useful to include that functionality so we can skip a certain number of records from a data stream in the future, if needed. The validator
argument is a function that accepts a single argument itself. We’ll use it to determine if the passed data
object argument is valid and should be included in the exported data set or not.
The only other significant addition is in the .on('data', ...)
function. We start by checking if there’s an offset
specified, in which case we want to skip that many records before processing begins.
Next we check if a validator
function was passed and, if so, whether the result of validating the current data
object is true or not. If the data
is valid, it is processed and added to a batch set as normal, otherwise we ignore that data
and increment the invalidCount
.
on('close', ...)
now outputs the total number of records processed, including those that were mutated, skipped, and considered invalid.
Open the gulpfile.ts
so we can adjust the db:generate:data
task to perform the validation rules discussed above.
gulp.task('db:generate:data', async () => {
try {
const query = `{
data(func: has(link_id)) {
link_id
}
}`;
// Get current comments
const comments = _.map(
(await new DgraphAdapter().query(query)).data,
comment => comment.link_id.substring(3)
);
const args = minimist(process.argv.slice(3), {
default: {
batchSize: 250,
limit: 1000,
offset: 0,
path: './src/data/RS_2018-02-01'
}
});
const stream = fs.createReadStream(args.path, {
flags: 'r'
});
// Include optional command line arguments in options.
const result = await DgraphAdapter.mutateFromStream(
Object.assign(
{
stream,
validator: (data: any) => {
if (
data.domain &&
!data.crosspost_parent &&
!data.over_18 &&
// If comments exist check that post contains at least 1.
data.num_comments > 0 &&
(comments && comments.length > 0
? _.includes(comments, data.id)
: true)
) {
// Posts
return true;
} else if (data.link_id) {
// Comments
return true;
}
return false;
}
},
args
)
);
console.log(result);
} catch (error) {
throw error;
}
});
As mentioned above, we’ll be adding comments to the database first, so the task starts by querying Dgraph and retrieving the full collection of comment nodes.
{
data(func: has(link_id)) {
link_id
}
}
curl http://127.0.0.1:8080/query -XPOST -d '
{
data(func: has(link_id)) {
link_id
}
}' | python -m json.tool | less
package main
import (
"context"
"flag"
"fmt"
"log"
"github.com/dgraph-io/dgraph/client"
"github.com/dgraph-io/dgraph/protos/api"
"google.golang.org/grpc"
)
var (
dgraph = flag.String("d", "127.0.0.1:9080", "Dgraph server address")
)
func main() {
flag.Parse()
conn, err := grpc.Dial(*dgraph, grpc.WithInsecure())
if err != nil {
log.Fatal(err)
}
defer conn.Close()
dg := client.NewDgraphClient(api.NewDgraphClient(conn))
resp, err := dg.NewTxn().Query(context.Background(), `blahblah`)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Response: %s\n", resp.Json)
}
As discussed in the Adding Comments to the State section Reddit prefixes link IDs with t3_
so the db:generate:data
task removes that prefix since each post node’s id
predicate does not contain that t3_
prefix.
Everything else in this task function is the same as before, except we’re passing the validator
property to the DgraphAdapter.mutateFromStream
method call. This validator function expects the single data
argument and performs the validation rules discussed above. For posts
we ensure they aren’t crossposts, that they aren’t NSFW, that they have at least one comment, and that at least one of those comments has a link_id
equal to the id
of the post record.
Update the Database Data Set
That’s all the validation logic we need, so now we just have to update the data set within the database to use our new validation rules.
Start by dropping the data with
gulp db:drop
and then reset the schema withgulp db:schema:alter
.$ gulp db:drop && gulp db:schema:alter [03:13:24] Requiring external module ts-node/register [03:13:25] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [03:13:25] Starting 'db:drop'... All Dgraph data dropped. [03:13:25] Finished 'db:drop' after 438 ms [03:13:26] Requiring external module ts-node/register [03:13:27] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [03:13:27] Starting 'db:schema:alter'... Dgrap schema altered. Dgrap schema altered. [03:13:27] Finished 'db:schema:alter' after 77 ms
Import comment data with
gulp db:generate:data
. We’ll be using alimit
of50000
from each file this time, but feel free to adjust this number to suit your needs.$ gulp db:generate:data --limit 50000 --path ./src/data/RC_2018-02-01 && gulp db:generate:data --limit 50000 --path ./src/data/RC_2018-02-02 [03:21:30] Requiring external module ts-node/register [03:21:32] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [03:21:32] Starting 'db:generate:data'... ████████████████████████████████████████ 100% | Elapsed: 1m45s | ETA: 0s | 50000/50000 records Processed 50000 records (50000 mutated, 0 invalid, 0 skipped). [03:23:17] Finished 'db:generate:data' after 1.75 min [03:23:18] Requiring external module ts-node/register [03:23:20] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [03:23:20] Starting 'db:generate:data'... ████████████████████████████████████████ 100% | Elapsed: 2m51s | ETA: 0s | 50000/50000 records Processed 50000 records (50000 mutated, 0 invalid, 0 skipped). [03:26:11] Finished 'db:generate:data' after 2.85 min
Lastly, import post data with a
limit
of5000
for each day.$ gulp db:generate:data --limit 5000 --path ./src/data/RS_2018-02-01 && gulp db:generate:data --limit 5000 --path ./src/data/RS_2018-02-02 [03:26:46] Requiring external module ts-node/register [03:26:48] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [03:26:48] Starting 'db:generate:data'... ████████████████████████████████████████ 100% | Elapsed: 1m34s | ETA: 0s | 5000/5000 records Processed 189813 records (5000 mutated, 184813 invalid, 0 skipped). [03:28:23] Finished 'db:generate:data' after 1.58 min [03:28:24] Requiring external module ts-node/register [03:28:26] Using gulpfile D:\work\dgraph\projects\dgraph-reddit\gulpfile.ts [03:28:26] Starting 'db:generate:data'... ██████████████░░░░░░░░░░░░░░░░░░░░░░░░░░ 34% | Elapsed: 2m34s | ETA: 5m5s | 1688/5000 records Processed 376678 records (1688 mutated, 374990 invalid, 0 skipped). [03:31:02] Finished 'db:generate:data' after 2.58 min
You’ll notice our validator is working as expected and skipping the majority of the post submission records because they fail validation. In fact, if you used the same data dump files as in this tutorial, you’ll see the second file doesn’t even contain enough valid post records to meet our
limit
of5000
. Instead, the entire file is processed with only about1700
valid submission to add.
In total this new import should take about 10 minutes to process but the end result is a much nicer data set within Dgraph! Try running the app again with yarn serve
and you should be able to click on any comments link within the front page post list to navigate to that discussion and see at least one comment. No more empty link views!