This is a cheeky little post I wrote in 2018 documenting how to use jq (not relation to jQuery) to parse over a GB work of JSON in a timely manner and without running out of RAM.
Head over to the Songkick Tech Blog and read about a way to go about solving this!
My first task when joining Songkick in 2017 was the implemantion of a new data ETL pipeline using go, python and the Google Cloud Platform.
Head over to this post in the Songkick Tech Blog to read about the highlights of this fantastic project!
Or Ruby, or Java. Heck, even perl! Our proud little app can integrate with any language that has a dnode implementation!
When installing the php dnode package with composer, I was surprised to find installed a React dependency that actually has nothing to do with facebook’s! It seems to be a great library for evented i/o for php.
Also, when I first attempted this experiment, I tried using Facebook’s php library based on v8 or straight php v8 but it soon became obvious that another solution was needed.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
dserver = dnode({
renderIndex: function(remoteParams, remoteCallback) {
var router = Router.create({
location: '/developers/page/' + remoteParams.page,
routes: routes
});
return router.run(function(Handler, state) {
gutil.log("Serving to PHP via " + dnodePort);
var component = state.routes[1].handler;
return component.fetchData(state.params, function(err, data) {
var reactOutput = React.renderToString(React.createElement(Handler, {
data: data,
params: state.params
}));
return remoteCallback(getHtml(reactOutput, data, state));
});
});
}
});
dserver.listen(dnodePort);
gutil.log("dnode: Listening on port " + dnodePort);
Explanation: These lines come after the declaration of our getHtml function in our server script (around line 48 over there). To begin things, we declare the name of the function we want to expose via dnode, in this case renderIndex. When the php side of dnode calls, it will provide the parameters and a remote function to be invoked as a callback once we’re done.
Inside our renderIndex function, on line 2, we’ll perform the now familiar task of getting a route handler for /developers/page/ (appending the page number sent by php). Then we go ahead and invoke the handler’s fetchData and use the resulting data to have React.renderToString provide the final html. Finally on line 18 we invoke the remote callback with the output of our handy getHtml function. Bam!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
<!DOCTYPE html>
<html lang="en">
<head>
<title>Sample PHP page rendering Node/ReactJS</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
</head>
<body style="background:#FAFAFA;padding:20px">
<div class="container">
<div class="jumbotron">
<h1>Sample PHP page</h1>
<p>The content below is rendered server-side with PHP/node/react via dnode.</p>
<p>Subsequent pagination is done with react client-side via Ajax.</p>
<p>This could be a Drupal, Wordpress, or any other kind of PHP app :)</p>
</div>
<div class="panel panel-default" style="padding:20px;">
<?php
require 'vendor/autoload.php';
$path = explode('/', $_SERVER['REQUEST_URI']);
$options = array('page' => array_pop($path));
$loop = new React\EventLoop\StreamSelectLoop();
$dnode = new DNode\DNode($loop);
$dnode->connect(3001, function($remote, $connection) use($options) {
$remote->renderIndex($options, function($result) use ($connection) {
echo $result;
$connection->end();
});
});
$loop->run();
?>
</div>
</div>
</body>
</html>
Explanation: The markup is just a simple html page, but close your eyes for a second and picture this implemented properly on a large, living app.
The magic starts at line 26, where we tell our dnode php instance to connect to port 3001, the one we defined in our node server script as dnodePort = 3001. This gives us a $remote object from which we can make calls to functions implemented on the other side of this funny wormhole, back in node.
So, when we do $remote->renderIndex that renderIndex is the function we just defined on our server script a couple of paragraphs above! When I saw this running the first time my mind was totally blown (away)! - yes, I know I need to go out more.
Next, we just pass the current page inside an $options array and get back a $result variable containing the output of the node app, including a link to our bundle.js that includes the browser-side react-routing code!
Our PHP app now serves pre-rendered react components! Once the page is ready, all user interaction is handled by the react components client side, communicating via ajax to the node app. Note that we can send the response from our node app in array so we can have body and head elements separately (like open graph tags, for example).
This ends this short series and I hope it was fun and useful for people learning node, gulp, react, and curious about remote procedure calls, etc.
Don’t forget to check out the repo and get in touch with suggestions, questions, or complaints about my cat sneaking into your balcony and causing mischief!
Now the plot begins to thicken. It’s time to work a little bit on the backend, behind the courtains.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
var React = require('react');
var Router = require('react-router');
var routes = require('./routes');
var dnode = require('dnode');
var express = require('express');
var app = express();
var gutil = require('gulp-util');
regularPort = 3000;
dnodePort = 3001;
app.get('/developers/page/*', function(req, res) {
var router = Router.create({
location: req.url,
routes: routes
});
return router.run(function(Handler, state) {
gutil.log("Serving to browser via " + regularPort);
var component = state.routes[1].handler;
return component.fetchData(state.params, function(err, data) {
var reactOutput = React.renderToString(React.createElement(Handler, {
data: data,
params: state.params
}));
return res.send(getHtml(reactOutput, data, state));
});
});
});
app.use(express["static"](__dirname + '/public'));
server = app.listen(regularPort, function() {
return gutil.log("HTTP: Listening on port " + regularPort);
});
getHtml = function(reactOutput, data, state) {
var response;
response = '<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">';
response += "<script>window.reactData = " + (JSON.stringify(data)) + "</script>";
response += "<script>window.initialPage = " + state.params.page + "</script>";
response += '<div id="app" class="container">';
response += reactOutput;
response += '</div>';
return response += "<script src='http://localhost:" + regularPort + "/js/bundle.js'></script>";
};
Feeble attempt at an explanation: Ok, let’s take this one step at a time. We start first by including our routes definition and setting up a simple Express app. Finally we declare two variables that will tell node and dnode on which ports they should listen for connections.
On our Express app instance, we will define a function to handle GET requests to /developers/page/*. This function returns a react-router instance created using the current req.url and our routes configuration (line 12). Next we invoke the run function on our router instance and obtain the react component that will handle the route and the current route state (params).
On line 22 we invoke the static function we coded into our DeveloperList component (FetchData, remember?) and pass a callback function to act upon the resulting data from the request to Github’s API.
On line 24 we invoke React’s renderToString method passing our DeveloperList component (represented by the Handler variable), the resulting data from fetchData, and finally the state.params provided by react router (page: 1 in this case). The resulting HTML we pass to a custom function called getHtml that will set the data as JSON object in the page so react can use it the first time. This function also includes bootstrap and more importantly, the bundle.js file generated by browserify. All this unwholesome HTML concatenatin’ can be replaced with a jade template for example, but just wanted to keep it all server stuff on a single file for this post :)
We wrap things up on line 28 by sending the output of our getHtml function as the response from our express app back to the browser.
We now have a working server and router to handle the initial request of a user. But we also need to configure our router to handle requests done once the page has been served, and do this without bothering the server script. We need now a browser script!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
var React = require('react');
var Router = require('react-router');
var routes = require('./routes');
Router.run(routes, Router.HistoryLocation, function(Handler, state) {
if (window.initialPage === parseInt(state.params.page)) {
return React.render(
<Handler data={window.reactData} params={state.params}/>, document.getElementById('app')
);
} else {
var component = state.routes[1].handler;
return component.fetchData(state.params, function(err, data) {
React.render(
<Handler data={data} params={state.params}/>, document.getElementById('app')
);
});
}
});
Explanation: Thanks Cthulhu and all the Great Old Ones, things get more simple from now on. Ish. The first three lines should look familiar by now, so we can move along. On line 5 we create once again a router instance but this time we pass the browser url provided by Router.HistoryLocation together with our routes definition.
We need to tell react to render the resulting Handler (our trusty developers list) using either the JSON data that comes already hardcoded on the page thanks to node, or to do a new call to Github’s API, in order to avoid doing an unnecessary request the first time. That little if on line 6 makes sure of just that. So, when it’s the first render of the page, we just invoke React.render with the data present, otherwise we make a call to the fetchData function of our Handler and use the result of that as the data to be rendered. I still feel there must be a better way to handle this scenario. Looking forward to improving on that!
That’s it! We have a this point an isomorphic/universal/$nextTrendyName node app, that makes use of the same react components on both the server and the client! It’s Sunday and it’s sunny outside, so time to head out and walk a bit in the sun. But first, let’s hook all of this up to PHP.
Because you only live once.
Now that we have coded our components, we need to link them up to a route handler, so when a user goes to http://localhost:3000/developers/page/1 we can catch that page 1 parameter and pass it as a property to our developers list.
1
2
3
4
5
6
7
8
var React = require('react');
var Route = require('react-router').Route;
module.exports = [
<Route>
<Route name='index' path="/developers/page/:page" handler={require('./developer_list.js')} ignoreScrollBehavior/>
</Route>
];
Explanation: At the top we declare our React dependency and then go ahead and create an instance of react-router. The next step is to export a route declaration, in which we declare that the path /developers/page/:page will be handled by the developers_list.js component.
The ignoreScrollBehavior is a handy little parameter that prevents the browser from scrolling up when a user clicks on our pagination links. The page paramater will later tickle down via state.params.page to the props object of our react components.
As a nice bonus of using React Router we have managed to create three state-less components. It’s considered a best practice to keep as little state as possible in your components, since every change of state triggers a new rendering of the component. As you can see, none of the components implement a getInitialState function and only work with the props handed down to them from above.
Later on, we’ll include this route definition script in our server script and browser script files in order to process urls coming via http requests and browser locations respectively.