Luis Elizondo Blog

Helping Education with Open Data in Mexico

I never blog about my job at IIIEPE and what we (or I) do here. I’ll do it know because I think it is relevant.

More than a year ago we created (and I coded it) a website made with Drupal: Coleccion SíaEducacion is a digital repository specialized in Education. We created it to solve one big problem, the local department of Education in the State of Nuevo Leon, which has more than 40,000 teachers, needs to constantly distribute several documents to those teachers in 4000+ schools; some of those documents are administrative documents, some of them have educational content. Before Colección SiaEducación, they were doing it with CDs and DVDs. With 4000+ schools you can imagine the size of problem.

With Coleccion SiaEducacion teachers now have a central location where they can locate and download the documents they need when they need them, usually in multiple and open formats, and the administrative authorities publish information when they need it. Also, any teacher who wants to contribute and publish documents can do it by requesting access.

Coleccion SiaE (as we call it at IIIEPE), saves hundreds if not thousands of hours a year burning CDs and DVDs. Saves hundreds of hours to the teachers locating a file inside 10 CDs with no search engine. Saves thousands of pesos ($) by eliminating the need to burn CDs and eliminating the shipping process at all. Finally, since we have a really good SEO (Search Engine Optimization) Google indexes everything and those documents are easier to find and are available to everyone out there who’s interested.

The website is running across 3 servers running on Linux, some of the technologies we use are Apache Solr as our Search Engine, and Memcache and Varnish to handle the traffic. Right now, the traffic is around 4000 page views a day. Is not a lot but is not a regular blog, our statistics say the traffic increases during school-time and we’re starting to see an increase in traffic if we compare it with last year when we launched.

I can’t finish without mentioning that we open sourced a few components back to the Drupal community while making Coleccion SiaE because at IIIEPE believe in Open Source.

Colección Sía Educación is an amazing project and we expect it to grow as more schools jump in and start using it. If you want more information, please contact me on Twitter @lelizondo

Holy sh$% Batman: showing async blocks of data from multiple sources with jQuery, Node JS and Express JS and the problem with Virtuals, DynamicHelpers and Partials

I’ve been using Node.js lately, and it has some problems like everything on planet Earth because it’s not perfect. I’ll give you an example and my solution:

You have a page, that shows a list of something that comes from the database. This is just main content, could be a list of blogs or something. On the sidebar you have totally unrelated (or related) content that also comes from the database. The problem with Node.js (especially using Express JS) is that you have to build the render object before you pass it to res.render which in turn will pass it to the template file (jade or whatever). I’ll explain the two ways of doing this:

Using async

Remember you have to query the database to get “somedata” and “somextradata” to pass both of them to res.render:

res.render(“myview”, {listone: somedata, listtwo: someextradata});

Using async you’ll do something like this:

Easy. You just executed two queries in series, first one, then the other one, and when both finish, you execute a last callback which is gonna have a results object with the data from each of the callbacks you executed in series. With that object (the results object) you can do whatever you want, like passing it to res.render(“myview”, results);

The problem with this approach is that you don’t really have lots of reusable code and you have to wait until all queries finish.

Using callback hell

Love it or hate it, callback hell works like this:

What happened there? We first do the first query, save the results in the var someData and after we have the results we do the second totatlly unrelated query, when we have the results from the second query we save it in the var someExtraData. Since we only have two queries, we build an object “results” that includes someData and someExtraData and we pass both results to callback or even to res.render(“myview”, results)

The problem is the code, no wonder why they call it callback hell. And again, both ways are async. So, you can’t show the page to the user unless everything is finished building.

But, again, back to the original question, what if someExtraData is just a meaningless peace of content, that kind of content you use just to populate the right side of your page? You don’t really want to have all those problems do you? And what if it is dynamic? What if you don’t have to actually wait for the first query to return?

Well, I’m still trying to find a good approach to solve this problem, in the meantime I’m using jQuery. Yes, jQuery.

This is my approach:

First, build the important data into the main content area like you would do if you didn’t need someExtraData.

Then, create another app.get route to fetch only the someExtraData. Something like this

With that you can query http://example.com/someExtraData and get a json object. This will be in a total different time from the main content. And with jQuery you just do:

Now, this approach works, and I think is pretty good, if you have any comments please let me know here will you?

Partials and dynamicHelpers (locals in 3.0)

Finally I don’t want to end without talking about those guys.

Partials are a thing of express/jade (don’t really know) in which you can reuse small peaces of jade code to do things you always do, the best thing is that it uses dynamic placeholders so you can pass inside jade a variable to the partial.

Something like this:

partial(“myMenu”, {menu: items})

Items will be a full collection of menu items. Inside the myMenu.jade partial file, you’re supposed to render “items”. The problem with partials is that partials need data but this data always come from the main .jade file, which in turn, comes from the same source. So, both the main.jade and partial.jade files use the same data source: res.render(“main”, {myData: myData})

Now, dynamicHelpers or locals in Express 3.0 are a different thing, those are like “global” variables. Let’s say you print the version of your site on every .jade file you have. You don’t want to do:

res.render(“somefile”, {version: getVersion}

everytime do you? Instead, you use dynamicHelpers to process the getVersion, and inside all your jade files you’ll have always #{version}. What it’s also nice is the fact that you have access to req and res arguments.

Virtuals

Virtuals in Mongoose are a special property of the object you query, the property doesn’t exists in your model, you create it “on-the-fly”. Let’s say you have a firstName and a lastName in your model, and you always want to use the fullName, the problem’s you don’t have it, you need to build it using firstName and lastName. You do that with virtuals.

The problem with virtuals and dynamicHelpers are almost the same, the async problem. Node.js won’t wait till dynamicHelpers/Virtuals query the database. Node.js already rendered the page to the user by the time you’re thinking about querying the database. So, whatever you query, it won’t show up in your page.

I will post the solution at Github: https://github.com/lelizondo/nodejs-async-blocks-example

So, what do you think?

Mongoose vs mongojs on Node.js

Quick post. I’ve been using Node.js a lot, to be honest, I love it, along with Node, I’ve been using MongoDB which, as you may know if you read my last post, doesn’t support joins.

My first choice on a project was to use mongojs because it was plain simple. It uses the same structure you use on MongoDB directly with a very few exceptions, the main one is the ability to pass a callback as an argument.

I moved really fast with mongojs, but then I hit a wall. Joins. Is just not possible to do it with MongoJS for the same reason you can’t do it with MongoDB, you have to use some sort of black magic to do them, and this is when Mongoose excells.

To be honest, I tried to avoid Mongoose because I thought it was hard to use, defining a model was just “a waste of time” when with MongoJS you don’t need to. I was wrong.

Bottom line. If your project is really simple, no joins, no complicated features, go with MongoJS, is really easy but limited. If you’re trying to save the world with your crazy idea and you need more powers than Superman, spend some time learning Mongoose and use it, it will take you there.

Trust me, I just spent two days rewriting everything using Mongoose, I wish someone told me before.

Joins on MongoDB with Mongoose and NodeJs

If you don’t know, non relational databases (NoSQL) can’t do joins, that thing that helps you query two different tables in one simple query, something like:

SELECT posts.title, users.name FROM {posts} LEFT JOIN posts.userid = users.userid ….;

Document-oriented databases, like MongoDB, allows you to easily create collections and documents, and I thought the best way to do “joins” was to embed the user object in the post.author property inside the post object, but this presents a big problem, what happens if inside the user object you have fields like username or picture that the user can change? An update operation on all the posts would be crazy, and for each collection that you have that has a user object embedded in the model you would have to do an update just to change the picture. Ridiculous isn’t it? That’s what I thought.

Then, I spent some time saving just the user _id into the posts.author property, and every time I wanted to query the posts I would just do two queries, first, get the post, then with the post.author get the user. That’s also expensive and not very clean in your code. And if you’re doing loops (which I try to avoid because of the async/sync thing on Nodejs), you could run into problems really fast.

Introducing Mongoose and populate

It seems that with Mongoose (v3) is really, and I mean really easy to do “joins”. Yeah, I double quote joins because they are not really joins but from a result-oriented perspective (is there another one?) that’s what you’re doing.

Mongoose comes with a feature called “populate”. Let’s explain:

Suppose you have two models, Users and Posts, I’ll keep things simple for now.

var UserSchema = new Schema({

  username: String

})

var PostSchema = new Schema({

  title: String,

  author: {type: Schema.Types.ObjectId, ref: “User”}

});

That’s easy, we just define two schemas and in author we create a property that will be a reference to the User schema. On save, we will save the _id of the user object.

The fun part comes when doing a findOne or find:

var id = “some id of the post we want to query”;

Post.findOne({_id: id})

  .populate(“author”)

  .exec(function(error, result) {

    // do something with the result like render it

  })

What is nice about this is that Mongoose will do a query on the users collection and get the author of the post and embed it in the result. You’ve just done a join on MongoDB my friend.

More information about populate here: http://mongoosejs.com/docs/populate.html

6 great Gnome 3 Shell Extensions

Using Gnome Shell or Gnome 3 (they are actually kind of the same thing)? Then you will love the following extensions:

All in one places

https://extensions.gnome.org/extension/299/all-in-one-places/

Shows a small button at the top bar with links to specific folders. You’ll love it if you miss the Places menu in Gnome 2

Alternative Status Menu (my favourite)

https://extensions.gnome.org/extension/5/alternative-status-menu/

This is the best one, because for some reason, Gnome removed some very useful buttons to power off, hibernate and suspend your machine. This one restores them.

Dash to dock (my second favourite)

https://extensions.gnome.org/extension/307/dash-to-dock/

The dock without you having to enter the activities workspace.

NetSpeed

https://extensions.gnome.org/extension/104/netspeed/

Shows the speed of your WLAN or ETH connection.

Tracker Search

https://extensions.gnome.org/extension/284/tracker-search/

This is another favourite, allows you to actually search documents from the Gnome Shell Search bar. Without this one, you can only search recent documents.

Show Desktop Button

https://extensions.gnome.org/extension/64/show-desktop-button/

If you’re using your Desktop as your traditional Desktop, where you can drop icons and documents, then you’ll need this one to quick access it.