Jetpack and the Infinite Scroll

Infinite Scroll
Everybody seems to be talking about Photon. Nobody really seemed to notice the Infinite Scroll feature added to the latest Jetpack. While it is slightly buggy, it’s pretty neat nevertheless. It’s also pretty easy to add support for in your own themes. I added it to this blog, just to see how. :)

So, here’s how you do it.

Step 1: Use Template Parts

Look at the HTML produced by your main Loop. The main one being the one that displays the main set of posts. This is usually in the index.php file.

Let’s say that HTML looks like this (overly simplified, of course):

<div id="content">
	<div class="post">...</div>
	<div class="post">...</div>
	<div class="post">...</div>
</div>

Each one of those posts is produced inside the loop, while the content container is outside it. Now let’s look at a simplified version of what The Loop might look like in PHP:

<?php while (have_posts()) : the_post(); ?>
        <div id="post-<?php the_ID(); ?>" <?php post_class(); ?>>
... all the post creation stuff here ...
        </div>
<?php endwhile; ?>

What we need to do is to separate that “post” from the Loop itself, in a way that the Infinite Scroll code can then produce the post independently. With Template Parts, this is actually rather easy. We just cut and paste all that post-producing code, and put it in a new “content.php” file. Then we change the Loop into this:

<?php 
while (have_posts()) : the_post();
	get_template_part( 'content', get_post_format() ); 
endwhile; 
?>

The get_template_part function will load that content.php file for us and include it there.

See a template is a file listed in the Template Hierarchy. But a template part is a file that contains only part of a template, and you can name them yourself. In this case, we named it “content”. It will become, therefore, content.php in the theme.

We also created a mini-hierarchy for ourselves while we were at it, using the Post Formats. What this means is that if I want to display, say, the “quote” format posts differently, I can copy content.php to content-quote.php, and make my changes in there. Then quotes will be displayed however I like in that new template part. And if content-quote.php doesn’t exist, then it falls back to using content.php, just like the normal fallbacks.

By separating out the Post from the surrounding loop that it’s in, then each post can be generated independently, sent over the wire via Javascript, and inserted into the page. Thus, Infinite Scroll can do the job.

Step 2: Enable Infinite Scroll

A theme can declare support for infinite scroll using add_theme_support(), like so:

add_theme_support( 'infinite-scroll', array(
	'container'    => 'content',
) );

The “container” is the ID of the element in the page that was our wrapper class. Let’s go back to that HTML again:

<div id="content">
	<div class="post">...</div>
	<div class="post">...</div>
	<div class="post">...</div>
</div>

See how the “id=content” is the big wrapper for our posts? Well, we tell the Infinite Scroll code that, it then uses it in the Javascript to know where to append the new posts to. Easy.

Step 3: Finishing touches

Simply speaking, that’s all there really is to it. But, there’s additional ways to customize it for your particular theme, to look better. Each of these is an option you can add to that add_theme_support call. Let’s go over them briefly:

Footer: You may have noticed when you enabled this and test it that you now have a sort of white footer overlay at the bottom of the page. This provides the page with a footer space that scrolls with the page, giving it a sort of finished look. Some people may like it, some not so much. We can use the footer option to adjust it.

First, using ‘footer’=>false will simply turn it off.

Alternatively, giving it the ID of some other element on the page will make it match that element’s width. For example, if you have HTML that looks like this:

<div id="page">
	<div id="content">
		<div class="post">...</div>
	</div>
	<div id="sidebar">
		<div>...</div>
	</div>
</div>

..then using ‘footer’=>’page’ would make it match the width of the “page” element, so as to be the proper width across the screen to fit better with the page.

Type: Infinite Scroll supports two methods of loading new posts. One method is called ‘scroll’, where new posts are loaded as the user reaches the bottom of the page. When they scroll, essentially. The other is called ‘click’, which adds in a new button towards the bottom of the posts, and when you click it, it loads more posts. You can force one type or the other using the ‘type’ option.

Footer Widgets: Some themes support having widget spaces in the footer. This has become fairly commonplace after Twenty Ten did it. Well, if you’re using the scroll type, then the widgets won’t be seen for a while on some blogs, since it will just load in more posts and keep pushing that footer further down. You could set the type to click to allow the footer to actually be seen.

The ‘footer_widgets’ option lets you specify a widget space that you registered with register_sidebar(). It will check that space to see if there’s any widgets in it. If there are, then it will automatically set type to “click” so that those widgets can be seen. If not, then it will let the type stay as “scroll”. Thus, it can choose the type intelligently, depending on whether widgets exist in the footer or not.

Wrapper: If you have some complex HTML for your posts, then you can give the ‘wrapper’ option a classname to use. It will wrap the returned posts in a div with that custom classname for you. This also allows you a way to perform styling tricks on those just-loaded posts.

Render: The ‘render’ option gives you an alternative to making a content template part. You can give this a function name, and that function will be called to render the posts instead. If you’re writing a theme, I suggest using the content template part instead. It’s easier and way more flexible.

Posts per page: By default, the Infinite Scroll module makes the page show 7 posts only, then more load when you scroll down (for the ‘scroll’ type). This is smaller than most people’s settings (default of which is 10), but it doesn’t matter since the whole point is to make it load new posts when you scroll down to them. If you happen to need a different number, you can use the ‘posts_per_page’ option to change the number.

There’s also a few things you can do with CSS.

When infinite scrolling is enabled, then body_class (which you should have been using in the body tag anyway) will get the “infinite-scroll” class added to it. If the type setting is ‘scroll’ instead of ‘click’, then the class “neverending” will be there too. Also, once the user gets to the last post, the javascript code adds an “infinity-end” class to the body, meaning that you can style what happens when the user gets to the end. Jetpack’s documentation page suggests hiding the normal footer until the user actually reaches it, using CSS like this:

/* Hides navigation links and site footer when infinite scroll is active */
.infinite-scroll #navigation,
.infinite-scroll.neverending #footer {
	display: none;
}
/* Shows the footer again in case all posts have been loaded */
.infinity-end.neverending #footer {
	display: block;
}

But obviously you can use it for more than just this functionality; because it’s on the body, the classes can be used to adjust anything on the page, and make them different with the infinite scroll than without it.

You can also style the “span.infinite-loader”. This is the spinning handle thing that shows up at the bottom of posts, before the new ones load in. Honestly, the loading is pretty quick and the default style looks pretty good to me. You can adjust the style of it if you like, perhaps to make it more centered on the container.

The div#infinite-handle will be the box that shows up to load more posts, if the type is set to “click”. You can use this to reposition the box, make it wider, change the color, etc.

Finally, if you are doing anything extremely tricky with Javascript, loading in new posts may be not handled precisely well by your existing code. The official support page methods a post-load event fired on document.body that occurs when the posts are loaded. You should refer to that documentation for more info on it, if needed.

Summary

On the whole, it’s a clever module to add to Jetpack. It still has some problems with specific sorts of content (some videos display very wonky sometimes), but I’m sure they’ll have that sorted out before too long. Adding support for it in your theme might also get you thinking about Template Parts and how to use them, and that’s definitely a good thing to be doing.

Using SFC? Got an email from Facebook?

Some people have been forwarding me this email message that they received from Facebook:

We currently detect that your app is using the old JavaScript SDK (FeatureLoader.js). This library will no longer work for authentication on February 1st, 2012 since it does not support OAuth 2.0. In May, we announced that all apps on Facebook need to support OAuth 2.0 by October 1st, 2011. Please upgrade to the new JavaScript SDK by February 1st, 2012 to avoid any disruption of service to your app.

The Simple Facebook Connect plugin has not used the FeatureLoader.js script since before version 1.0, which was released 5 months ago. Version 1.2 of SFC fully integrated OAuth 2.0 authentication, and it was released 5 weeks ago.

So if you’re getting this email from Facebook, upgrade SFC to the latest version. Problem solved.

Migrating an SFC Application to a Fan Page

Still getting emails about this one, so here’s a quick rundown on how to do it.

First, if you were already using a Fan Page, then you are not affected at all and don’t have to do anything. Please stop emailing me and asking for confirmation. Thanks. :)

Now, if you were using your Application’s Wall as your Page (like I was doing and even recommended), then Facebook is killing off the “Wall” of your Application. This is not a big deal, actually, and you can migrate your Fans to a new Page rather easily.

Step One: Create a new Page. Visit this page to do so. Note: You MUST select “Brand or Product”, and in the dropdown you MUST select “App”. This is not optional. You have to do this to migrate your Fans.

Also note that you must make the name of the Page EXACTLY THE SAME as the name of the application. This is important, don’t try to rename your stuff yet.

Step Two: After you’ve created the page, you’ll want to connect it to your site (using SFC, naturally). First, get the ID number of your new Page. You can find this in the URL of the “Edit Page” link on that Facebook Page. Once you have the ID number, put it into the “Facebook Fan Page” field on the SFC Settings screen and save. While you’re on this Edit Page link on Facebook, you can upload your logos, configure it, etc. Note: Do NOT select a new Vanity URL. The migration will migrate your old one if you had one.

Step Three: Configure SFC. If you’re using the Publisher, for example, you may have to click the grant permissions button again to have it get the new access token for the page. You may need to turn on auto-publishing to the page. Stuff like that. For the most part, SFC is pretty good at configuring itself for this, the Fan Box will automagically switch over, etc.

Step Four: Test. Make a new Post and see if it publishes to your Page. Try the Manual Publisher boxes. Verify that it’s working, basically. While you’re at it, you might go and manually publish some of your older posts to the Page, since the migration will not migrate the content on the wall.

Step Five: Migrate. Visit your application’s profile page. If you don’t see the box below, wait a day or two and it will eventually appear:

Use that migrate link and you’ll get a popup box allowing you to select a Page.

WARNING: If you get a popup that says “You don’t have any eligible Facebook Pages to migrate to”, then STOP RIGHT NOW. Do NOT click migrate. You only get one chance at this, if you mess it up then it’s broken forever.

If you have a Page, and it’s a “Brands or Products/App” page, and it has the EXACT same name as your Application, then you will be given a dropdown to select that Page. Otherwise, you’ll get the bad message. Click Cancel in such a case, fix your Page, and then try again. Only when you have the dropdown and have selected your page should you continue.

Step Six: Patience. Once you’ve selected your new Page and clicked Migrate (and remember, you only get one shot at this!), then after a while, a few things will happen:

a) Your Fans of the Application will slowly be migrated to be Fans of the new Page instead.

b) If you had a vanity URL on the Application Page and did not have one on the Fan Page, then the vanity URL will get migrated too.

c) Your Application Wall will disappear forever (this happens instantly) and any links to it will redirect to your Fan Page.

And that’s it. You’re done. Works fine with SFC. The next version of SFC will remove the publishing to Application Pages entirely, as well as the (now misleading) wording.

 

SFC and New Facebook Changes

Facebook is getting rid of Application Profile Pages, and allowing people who are using them to transfer their subscribers to normal Fan Pages. SFC will be changing soon to adapt to this change, but the existing Fan Page support in SFC works fine and can be used right now.

I’ve tested out this migration process on one of my pages, and it works fine. Here’s what you have to do to make it work with SFC if you were not using a Fan Page already (note, if you were using a Fan Page already, then you’re done and must change nothing at all).

1. Create a new Fan Page in the Brands/Product -> App category.

2. Give it the same name as your App (exactly the same, mind you).

3. Set up the new fan page however you like. Take its ID number and put that into SFC, then use the Manual publisher to fill out the wall with some of the older posts (the wall content will NOT be migrated when you do the migration).

4. When you visit your app’s wall, you’ll get the migration message (eventually). You can use this to migrate all the people who have liked your application to having liked the new Fan Page. If you used a vanity URL, this will transfer also *if* you don’t put a vanity URL on the Fan Page.

After you’ve migrated the likes and changed SFC to be publishing to the Page, you can continue on as normal. Nothing else about SFC changes. Since Facebook will be eliminating App Profile Walls entirely in February, I’ll be removing support for them from SFC entirely before then. Expect that change to be in SFC 1.3.

How the Postname Permalinks in WordPress 3.3 Work

So, I first wrote about this topic on the wp-hackers list back in January 2009, explaining some of the scaling issues involved with having ambiguous rewrite rules and loads of static Pages in WordPress. A year later the same topic came up again in the WPTavern Forums, and later I wrote a blog post about the issue in more detail. That post generated lots of questions and responses.

In August 2011, thanks to highly valuable input from Andy Skelton which gave me a critical insight needed to make it work, and with Jon Cave and Mark Jaquith doing testing (read: breaking my patches over and over again), I was able to create a patch which fixed the problem (note: my final patch was slightly over-complicated, Jon Cave later patched it again to simplify some of the handling). This patch is now in WordPress 3.3.

So I figured I’d write up a quick post explaining the patch, how it works, and the subsequent consequences of it.

Now you have two problems.Quick Summary of the Problem

The original underlying problem is that WordPress relies on a set of rules to determine what type of page you’re looking for, and it uses only the URL itself to do this. Basically, given /some/url/like/this, WordPress has to figure out a) what you’re trying to see and b) how to query for that data from the database. The only information it has to help it do this is called the “rewrite rules”, which is basically a big list of regular expressions that turn the “pretty” URL into variables used for the main WP_Query system.

The user of the WordPress system has direct access to exactly one of these rewrite rules, which is the “Custom Structure” on the Settings->Permalink page. This custom string can be used to change what the “single post” URLs look like.

The problem is that certain custom structures will interfere with existing structures. If you make a custom structure that doesn’t start with something easily identifiable, like a number, then the default rewrite rules wouldn’t be able to cope with it.

To work around this problem, WordPress detected it and uses a flag called “verbose_rewrite_rules”, which triggers everything into changing the list of rules into more verbose ones, making the ambiguous rules into unambiguous ones. It did this by the simple method of making all Pages into static rules.

This works fine, but it doesn’t scale to large numbers of Pages. Once you have about 50-100 static Pages or so, and you’re using an ambiguous custom structure, then the system tends to fall apart. Most of the time, the ruleset grows too large to fit into a single mySQL query, meaning that the rules can no longer be properly saved in the database and must be rebuilt each time. The most obvious effect when this happens is that the number of queries on every page load rises from the below 50 range to 2000+ queries, and the site slows down to snail speed.

The “Fix”

The solution to this problem is deeper than simple optimizations. Remember that I said “WordPress relies on a set of rules to determine what type of page you’re looking for, and it uses only the URL itself to do this”. Well, to fix the problem, we have to give WordPress more input than just the URL. Specifically, we make it able to find out what Pages exist in the database.

When you use an ambiguous custom structure, WordPress 3.3 still detects that, and it still sets the verbose_page_rules flag. However, the flag now doesn’t cause the Pages to be made unambiguous in the rules. Instead, it changes the way the rules work. Specifically, it causes the following to happen:

  1. The Page rules now get put in front of the Post rules, and
  2. The actual matching process can do database queries to determine if the Page exists.

So now what happens is that the Page matching rules are run first, and for an ambiguous case, they’ll indeed match the Page rule. However, for all Page matches, a call to the get_page_by_path function is made, to see if that Page actually exists. If the Page doesn’t exist in the database, then the rule gets skipped even though it matched, and then the Post’s custom structure rules take over and will match the URL.

The Insight

The first patch I made while at WordCamp Montreal used this same approach of calling get_page_by_path, but the problem with it was that get_page_by_path was a rather expensive function to call at the time, especially for long page URLs. It was still better than what existed already, so I submitted the patch anyway, but it was less than ideal.

When I was at WordCamp San Francisco in August, hanging around all these awesome core developers, Andy Skelton commented on it and suggested a different kind of query. His suggestion didn’t actually work out directly, but it did give me the final idea which I implemented in get_page_by_path. Basically, Andy suggested splitting the URL path up into components and then querying for each one. I realized that you could split the path up by components, query for all of them at once, and then do a loop through the URL components in reverse order to determine if the URL referred to a Page that existed in the database or not.

So basically, given a URL like /aaa/bbb/ccc/ddd, get_page_by_path now does this:

SELECT ID, post_name, post_parent FROM $wpdb->posts WHERE post_name IN ('aaa','bbb','ccc','ddd') AND (post_type = 'page' OR post_type = 'attachment')

The results of this are stored in an array of objects using the ID as the array keys (a clever trick Andrew Nacin pointed out to me at the time).

By then looping through that array only once with a foreach, and comparing to the reversed form of the URL (ddd, ccc, bbb, aaa) you can make an algorithm that basically works like this:

foreach(results as res) {
  if (res->post_name = 'ddd') {
    get the parent of res from the results array
     (if it's not in the array, then it can't be the parent of ddd, which is ccc and should be in the array)
    check to make sure parent is 'ccc',
    loop back to get the parent of ccc and repeat the process until you run out of parents
  }
}

This works because all the Pages in our /aaa/bbb/ccc/ddd hierarchy must be in the resulting array from that one query, if /aaa/bbb/ccc/ddd is a valid page. So you can quickly check, using that indexed ID key, to see if they are all there by working backwards. If they are all there, then you’ll eventually get to parent = zero (which is the root) and the post_name = ‘aaa’. If they’re not there, then the loop exits and you didn’t find the Page because it doesn’t actually exist.

So using this one query, you can check for the existence of a Page any number of levels deep fairly quickly and without lots of expensive database operations.

Consequences

There are still some drawbacks though.

In theory, you could break this by making lots and lots of Pages, if you also made their hierarchy go hundreds of levels deep and thus make the loop operation take a long time. This seems unlikely to me, or at least way more unlikely than somebody making a mere couple hundreds of Pages. Also, WordPress won’t let you use the same Page name twice on the same level, so you’d really have to try for it to make this take too long.

If you try to make a URL longer than around 900K or so, the query will break. Pretty sure it’d break before that though, and anyway most people can’t remember URLs with the contents of a whole book in them. ;)

This also adds one SQL operation to every single Post page lookup. However, this is still better than having it break and try to run a few thousand queries every time in order to build rewrite rules which it can’t ultimately save. And the SQL being used is relatively fast, since post_name and post_type are both indexed fields.

Basically, for the very few and specific cases that had the problem, the speedup is dramatic and immediate. For the cases that use unambiguous rules, nothing has changed at all.

There’s still some bits that need to be fixed. Some of the code is duplicated in a couple of places and that needs to be merged. The pagename rewrite rule is a bit of a hack to avoid clashing, but it works everywhere even if it does make the regexp purist groan with dismay (for critics of this, please know that I did indeed try to do this using a regexp comment to make the difference instead of the strange and silly expression, but it doesn’t work because the regexp needs to be in a PHP array key).

Anyway, there you have it. I wrote the patch, but at least 5 other core developers contributed ideas and put in the grunt work in testing the result. A lot of brain power from these guys went into what is such a small little thing, really. A bit obscure, but I figured some people might like to read about it. :)