Category Archives: webdev

Chrome, HTML5, JavaScript, mobile, php, webdev

Modern Web Apps at Scale With Google App Engine (Part 1 Out Of 3)

modern web apps exampleThere is no secret that today any developer that wish to built the next G+, Path, Instagram etc’ must think on two major aspects:

  • Server Side / Cloud Service that she is going to use in order to create the API. Here we must answer some important questions like:
          1. What will I do if I get a huge spike in traffic?
          2. Will I need to manage it? Do I have to do it?
          3. How will I communicate with different clients?
          4. Which technology to use in order to store: data, state etc’
  • Client side technology
    • Web technologies: HTML5, CSS3
    • Mobile native app: Android, iOS and Windows.
In this post I’ll be focusing on the client side (and later this summer I’ll have another few posts on the server side) and what we should use in order to build a modern web application.

Modern Web App

It’s not an easy definition since web technology is running very fast. Nevertheless, we can find certain features across successful web apps:

  • Self Contained & Functional– They all have one specific goal and they do their best to provide the users the functionality to get to her goal. Few examples:
    • New York Times – Consume news.
    • Hipmunk – Find a flight that is the ‘perfect’ for your needs.
    • Gojee – Find the recipe you would taste and say WOW after it.
  • “Offline first” – You will want your web app to work offline. It’s an important feature that will let your users be productive on places like: planes, subways and on spots like: Starbucks when you don’t have (always) good connection. Another very important benefit will be the improve in performance. Since the app will work locally (first) and then sync the state/data the users will get responsiveness that are far better from an app that need to ‘wait’ for network on every action that the user does.
  • Client Side Architecture – Since we are moving big parts of our ‘logic’ to the client we need to think about methods that will keep our code readable and maintainable. I guess this is the main reason why we see so many MVC Frameworks. The only advice I can give here is to try few of the popular ones and see how they approach the separation between the data and the UI. If you have some time go over The Top 10 Javascript MVC Frameworks Reviewed. Then, after you try ‘some’ you will be able to pick the one that fit the bill in the best way. For the busy developer (I know… most of us don’t have too much free time 🙂 – Go and play with these three MVC (some will say it should be MV* because it’s not really MVC more MVVM, MVP):
ember.jsEmber.js – Don’t waste time making trivial choices angular.js Angular.js -Lets you extend HTML vocabulary for your application backbone.jsBackbone.js – gives structure to web applications by providing models with binding, collections and views
  • Device Aware – We all know that mobile is going ‘up and right’. So when you design your web app you should think on progressive enhancement and how it will fit  to different screen sizes. Good examples to look at are: Gmail, Flipboard and Twitter. If you want to go deeper on this interesting subject there is a webcast on ‘mobile web apps’ I did with Oreilly three weeks ago. You can go over the slides as well.

Challenges and Solutions

Challenge: What to do first (mobile app, web app, both). Focus is the most important thing for a startup so what should we do?
Solution: Built on the server a RESTful API that allow you to consume it from any platform. The way to work with an API is similar (more or less) to all the platforms although in the web we have some interesting new tools that we can use. If you are working with Google APIs here is a new API Explorer that is very useful to browse the capabilities of each API and to try it live.
Challenge: How to make the web app functional with clear goal?
Solution: Invest time and effort in your UX (and later the UI). Some taking it a step further and say that you should focus only on great UX and all the rest will follow. Who said apple?
Challenge: How should I work with RESTful APIs (in our case – google ones)
Solution: The first excellent tool will be Google JavaScript Client Library  Why it’s so powerful?
  1. It save us the trouble to reinvent the wheel and built simple functionality like: CRUD operations, list, order, search etc’. It’s all baked in it.
  2. It provide us some powerful new features:
    1. RPC Batch
    2. CORS
    3. Authentication out of the box
    4. Version control
    5. Super Simple
Challenge: How to make my application ‘offline first’?
Solution: With HTML5 we have few APIs that let us create web apps that will work great when there is no connection. The first step is to pretend that there’s no internet connection. It will force you to implement a sync layer that works only when online. So you will give the users the ability to add/edit/remove data and when the connection is online your app logic will do the syncing with the server. In order to have our app offline we should use two important features:
  1. Storing (static) assets: we can use AppCache. It’s our ability to save locally our html, js, css files and all the images, sound files etc’.
  2. Storing data: localStorage, IndexedDB, File API. This is a hot (and large) topic. I would suggest to read deeper on when and where to use each over at html5rock.com

webIntent

Challenge: There are so many web services I would love to hook into my app – How can I do it without reinventing the wheel each time? In other words, I want to give my users the ability to edit photos, share on twitter, g+ and Linkedin (just to name few).
Solution: WebIntent! If you are familiar with the intent system that is flourishing in Android you know what I’m talking about. We now have a powerful solution for the web. Web Intents is a framework for client-side service discovery and inter-application communication. The code you need to add to your app is as simple as:

var sharingDetails = "Check out my....";
var intent = new Intent(
            "http://webintents.org/share",
            "text/uri-list",
             beerDetails);
window.navigator.startActivity(intent);

The best part is that it will work on most browsers with JavaScript shim and in Chrome (19+) it’s built into the browser natively!

Google Chrome FrameChallenge: What can I do on old browsers that do not support HMTL5 very well?
Solution: Google Chrome Frame is an open source plug-in that seamlessly brings Google Chrome’s open web technologies and speedy JavaScript engine to IE. You can add it to your web app with one line of meta tag: <meta http-equiv=”X-UA-Compatible” content=”chrome=1″> or configure the server to send this http header: X-UA-Compatible: chrome=1 in both scenarios your users will enjoy the wonderful world of chrome (v8, HTML5, CSS3, WebGL etc’) inside their IE. Important fact to remember is that your users do not need admin rights (on their windows PC) in order to install it.

I hope this gave you some points to think about during the planning and designing phase of your next web app. For startups and developers that have an app in production, I would suggest to check what are the building blocks that will make sense to implement first (e.g. web intent, clean the client code by refactoring it to an MVC framework etc’).
This is the first post in a series of posts, I hope to cover Google App Engine in the next one and then combine the two worlds in the last post that will be more of a ‘cookbook’ to best practices of leveraging Google App Engine and HTML5.
Standard
JavaScript, webdev

JavaScript Maintainable Code

It’s a known fact that you read (=maintain) your code a lot more then you are writing it. Thus, you should be kind to your ‘future’ self (as the creator of SASS said very wisely).

In this presentation, Nicholas Zakas (from YUI fame) is doing it again with some powerful thoughts on how to keep your code in a shape you can do things with it when times comes (in most cases, it’s 5 minutes after you are ‘done’ with the beta version).

It’s full of good points. Some are basic (and importnat) like: Use CSS for the presentation layer, JavaScript to the behavior of your application and HTML for the structure (and data) and don’t cross the streams by making short cuts like putting some JS code inside you HTML and vis versa. He also talk about style (spaces vs tabs) and code readability. I’ve read his slides two times just to enjoy the cool pictures he is using… Plus it’s full of great tips that will put your code in a better shape.

Standard
Chrome, HTML5, JavaScript, webdev

Web Workers And Big Data – A Real World Example

Web Workers in the 19th centery

I had an interesting conversation on G+ with developers around web workers and the need to a ‘real world’ example that use ‘big chunk’ of data. In this example we will dive into this senario. No more, ‘hello world’ and calculation of some nice number (Pi, e etc’). In the code we will take an array of 33.55 millions numbers (~=32MB) make some calculation on each and everyone of them and return the new data to our main page/app. We will use  transferable objects because they are a new powerful option to ‘move’ (not copy) big arrays in and out the worker. Since the support for transferable objects is done with: webkitPostMessage() (so far). We will use Chrome as the browser for these examples.

This is the main page of our example. In the code snippert below you can see the test function that let us choose the method of delivery.


// The entry point for our comparing. 
function test(useIt) {
  // useIt: true  - use transferrable objects
  //        false - COPY function for sending the data. 
  var useTransferrable = useIt;
  setupArray(); // create the 32M array with numbers

  if (useTransferrable ) {
    console.log ("## Using Transferrable object method on size: " +
                 uInt8View.length);
    // This is the syntax to send by using trans-obj.
    worker.postMessage(uInt8View.buffer, [uInt8View.buffer]);
  } else {
    console.log ("## Using old COPY method on size: " + 
                 uInt8View.length);
    // Simple send msg that copy the data to the worker
    worker.postMessage({'copy': 'true', 
                      'ourArray': uInt8View.buffer});
  }
}

and here is the worker that is doing the hard work on 32M of numbers. You can think on better ways to do ‘hard work’… Specially if you are in the world of WebGL and you have big matrix under your arms.


  // Here we are 'computing' something important on the data. 
  // In our case - just play with % if you have new hardware
  // try: Math.sqrt( uInt8View[i] * Math.random() * 10000);
  for (var i=0; i < dataLength; i++ ) {
    uInt8View[i] = uInt8View[i] % 2;
  }
  
  if (useTransferrable) {
    self.postMessage(uInt8View.buffer, [uInt8View.buffer]);
  } else {
    self.postMessage(e.data.ourArray);
  }

The results are clear (as the sun over the beach in Maui). It was much faster to work with transferrable objects.

web workers - compare options to move data in and out
With transferrable objects it took us 292ms while with copy it was 533ms.
Last but note least, you can find more examples and deep coverage on web workers in my book on web workers. Psst… if you can’t sleep, it might help you on that front as well.
Web Workers - The book

Standard
Chrome, JavaScript, webdev

Web Intents – The Next Wave Of Web Apps

New tool set for web developersWhat?

Web Intents is a framework for client-side service discovery and inter-application communication.

Why and when?

As a web developer you can think on many cases where you’re building a web app and you want to allow users to share pictures (or edit or any other operation). You have few options:

  • Write that functionality yourself, but it will require a huge amount of work and likely won’t be nearly as good as other existing photo web apps.
  • Force a hard-code integration with a collection of existing photo editing web apps, but that can lead to a cluttered interface and requires manual intervention later to integrate with new photo editing apps that may emerge. Worse, it forces your chosen integrations upon your end users.

One of the greatest strengths of the web is that the ease of linking allows innovative new apps to succeed without asking anyone else’s permission–but up until now that hasn’t applied to integrations between web apps. Web Intents is an emerging W3C specification inspired by Android’s Intents system that aims to solve the problems of communications.

How does it work?

  1. Service registers its intention to handle an action for the user.
  2. App requests to start an action (share, edit, pick, view etc’).
  3. User selects which service to handle the action.

Here as some slides that explain the main concepts from a pervious talk I gave. For the ones that like short informative videos – Paul Kinlan (=the godfather of web intents!) intreduce you to the powerful world of web intent in 3 minutes.

In case you are wondering what can you do with this good stuff on other browsers the answer is that there is a limited shims exist, but I recommend feature detection to support browsers that don’t have Web Intents support.

Standard
Chrome, HTML5, JavaScript, webdev

IndexedDB Changes – SetVersion Is Out

Well, things are moving fast… We have now a new version for handling updating schema in IndexedDB. As you can see both MDN (mozilla) and Chrome/Chromium are aligned with the new way.

Here is a short example that show you the new way to work with upgrades to your schema:


<html>
<head>
<script>
var indexedDB = window.indexedDB || window.webkitIndexedDB
|| window.mozIndexedDB || window.msIndexedDB;
var request = indexedDB.open("testUpgrade",1);
var customerData=[
{tel:"540-343-3334", name:"Monk", age:35, email:"mock@example.com"},
{tel:"650-450-5555", name:"Jack", age:32, email:"jack-2@example.com"}
];
request.onerror = function(e){
console.log("Oppss… we got into problems and errors. e:" + e);
};
request.onupgradeneeded = function(event) {
console.log("UPGRADE our nice example DB") ;
var objectStore = db.createObjectStore("customers",{keyPath:"tel"});
objectStore.createIndex("name","name",{unique:false});
objectStore.createIndex("email","email",{unique:true});
for(var i in customerData){
objectStore.add(customerData[i]);
}
};
request.onsuccess = function(e) {
console.log("life is good with indexedDB e:" + e) ;
};
</script>
</head>
</html>

view raw

Test-indexedDB-api-changes.html

hosted with ❤ by GitHub

The full example I have on github that show the differences between WebSQL and indexedDB still uses the setVersion syntax. However, in Firefox 10 (and in Chrome 21) you should change your code to work with the new API. Good luck!

=====

[Update Sep 2012]

In case you wish to see a code that will work fine both on Chrome (that still need to update according to the spec on onupgrade) and Firefox you can check this example of jQuery mobile and indexedDB mobile app:


<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Short example on using indexedDB with jquery mobile – last updated: May 2012">
<meta name="author" content="Ido Green">
<title>IndexedDB with JQM</title>
<link rel="stylesheet" href="http://code.jquery.com/mobile/1.1.1/jquery.mobile-1.1.1.min.css&quot; />
<script src="http://code.jquery.com/jquery-1.7.1.min.js"></script&gt;
<script src="http://code.jquery.com/mobile/1.1.1/jquery.mobile-1.1.1.min.js"></script&gt;
<script>
var dbName = "jqm-todo";
var dbVersion = 1.0;
var todoDB = {};
var indexedDB = window.indexedDB || window.webkitIndexedDB ||
window.mozIndexedDB;
if ('webkitIndexedDB' in window) {
window.IDBTransaction = window.webkitIDBTransaction;
window.IDBKeyRange = window.webkitIDBKeyRange;
}
todoDB.indexedDB = {};
todoDB.indexedDB.db = null;
$(document).bind('pageinit', function() {
console.log("– lets start the party –");
todoDB.indexedDB.open();
$("#addItem").click(function() {
addTodo();
});
});
todoDB.indexedDB.onerror = function(e) {
console.log(e);
};
todoDB.indexedDB.open = function() {
var request = indexedDB.open(dbName, dbVersion);
request.onsuccess = function(e) {
console.log ("success our DB: " + dbName + " is open and ready for work");
todoDB.indexedDB.db = e.target.result;
var db = todoDB.indexedDB.db;
if (db.setVersion) {
console.log("in old setVersion: "+ db.setVersion);
if (db.version != dbVersion) {
var req = db.setVersion(dbVersion);
req.onsuccess = function () {
if(db.objectStoreNames.contains("todo")) {
db.deleteObjectStore("todo");
}
var store = db.createObjectStore("todo", {keyPath: "timeStamp"});
var trans = req.result;
trans.oncomplete = function(e) {
console.log("== oncomplete transaction ==");
todoDB.indexedDB.getAllTodoItems();
}
};
}
else {
todoDB.indexedDB.getAllTodoItems();
}
}
else {
todoDB.indexedDB.getAllTodoItems();
}
}
request.onupgradeneeded = function(e) {
console.log ("Going to upgrade our DB");
todoDB.indexedDB.db = e.target.result;
var db = todoDB.indexedDB.db;
if(db.objectStoreNames.contains("todo")) {
db.deleteObjectStore("todo");
}
var store = db.createObjectStore("todo",
{keyPath: "timeStamp"});
todoDB.indexedDB.getAllTodoItems();
}
request.onfailure = todoDB.indexedDB.onerror;
request.onerror = function(e) {
console.error("Well… How should I put it? We have some issues with our DB! Err:"+e);
}
};
todoDB.indexedDB.addTodo = function(todoText) {
var db = todoDB.indexedDB.db;
var trans = db.transaction(['todo'], "readwrite");
var store = trans.objectStore("todo");
var data = {
"text": todoText,
"timeStamp": new Date().getTime()
};
var request = store.put(data);
request.onsuccess = function(e) {
todoDB.indexedDB.getAllTodoItems();
};
request.onerror = function(e) {
console.error("Error Adding an item: ", e);
};
};
todoDB.indexedDB.deleteTodo = function(id) {
var db = todoDB.indexedDB.db;
var trans = db.transaction(["todo"], "readwrite");
var store = trans.objectStore("todo");
var request = store.delete(id);
request.onsuccess = function(e) {
todoDB.indexedDB.getAllTodoItems();
};
request.onerror = function(e) {
console.error("Error deleteing: ", e);
};
};
todoDB.indexedDB.getAllTodoItems = function() {
var todos = document.getElementById("todoItems");
todos.innerHTML = "";
var db = todoDB.indexedDB.db;
var trans = db.transaction(["todo"], "readwrite");
var store = trans.objectStore("todo");
// Get everything in the store;
var keyRange = IDBKeyRange.lowerBound(0);
var cursorRequest = store.openCursor(keyRange);
cursorRequest.onsuccess = function(e) {
var result = e.target.result;
if(!!result == false)
return;
renderTodo(result.value);
result.continue();
};
cursorRequest.onerror = todoDB.indexedDB.onerror;
};
function renderTodo(row) {
var todos = document.getElementById("todoItems");
var li = document.createElement("li");
var a = document.createElement("a");
var t = document.createTextNode(row.text);
a.addEventListener("click", function() {
todoDB.indexedDB.deleteTodo(row.timeStamp);
}, false);
// some fun with jquery mobile data attributes
a.setAttribute("href", "#");
a.setAttribute("data-iconpos", "notext");
a.setAttribute("data-role", "button");
a.setAttribute("data-icon", "delete");
a.setAttribute("data-inline", "true");
li.appendChild(a);
li.appendChild(t);
todos.appendChild(li)
// And lets create the new il item with its markup
$("#todoItems").trigger('create');
}
// Add an item only if we have more then zero letters
function addTodo() {
var todo = document.getElementById("todo");
if (todo.value.length > 0) {
todoDB.indexedDB.addTodo(todo.value);
todo.value = "";
}
}
// use it in case you wish to work on specific 'set' of data
function showAll() {
document.getElementById("ourList").innerHTML = "" ;
var request = window.indexedDB.open(dbName);
request.onsuccess = function(event) {
// Enumerate the entire object store.
var db = todoDB.indexedDB.db;
var trans = db.transaction(["todo"], IDBTransaction.READ_ONLY);
var request = trans.objectStore("todo").openCursor();
var ul = document.createElement("ul");
request.onsuccess = function(event) {
var cursor = request.result || event.result;
// If cursor is null then we've completed the enumeration.
if (!cursor) {
document.getElementById("ourList").appendChild(ul);
return;
}
var li = document.createElement("li");
li.textContent = "key: " + cursor.key + " => Todo text: " + cursor.value.text;
ul.appendChild(li);
cursor.continue();
}
}
}
</script>
</head>
<body>
<div data-role="page">
<div data-role="header">
<h1>IndexedDB with JQM</h1>
</div>
<!– /header –>
<div data-role="content">
<p>
This is a short example of inexedDB with jQueryMobile on a todo list app. Please open Chrome DevTools and/or FireBug in order to see all the log message and understand what is the process.
</p>
<p>
<input type="text" id="todo" name="todo" placeholder="What do you need to do?" />
<input type="submit" value="Add Todo Item" id="addItem" />
</p>
<ul id="todoItems" data-role="listview" data-inset="true" data-filter="true"></ul>
</div>
<!– /content –>
<div data-role="footer">
<p>
<ul>
<li>
<a href="https://greenido.wordpress.com">Ido's blog</a>
</li>
<li>
<a href="http://www.w3.org/TR/IndexedDB/">IndexedDB spec on w3c</a>
</li>
<li>
<a href="https://github.com/greenido/WebSQL-to-IndexedDB-example">WebSQL to IndexedDB example on github</a>
</li>
</ul>
</p>
</div> <!– /footer –>
</div> <!– /page –>
</body>
</html>

view raw

jqm-indexedDB-example.html

hosted with ❤ by GitHub

 

Standard
Chrome, HTML5, webdev

New USB API & Bluetooth API In Chrome/ChromeOS

It seems that we are going to have some powerful new APIs on ChromeOS and Chrome in the future. From looking at chromium site last month I’ve saw two new interesting proposal to new APIs that will make Chrome (even) better. The ability to ‘talk’ with hardware and external devices is very important and until today the way to do it (from a web app) was by using network. So there was no (real good) option to communicate with hardware and external accessories that do not support network (e.g Wi-Fi). These two new APIs are going to allow web developers with more power to build amazing apps that communicate with external devices. Think, smart watches, GPSs, robots, Lego cars (my kids will love it!) etc’.

USB API

The USB API aims to provide access to fundamental low-level USB operations from within the context of an extension. Some use cases that might come to mind: GPS, Watch, mobile phone or any other devices which require third-party drivers to work. One of the use cases for this API would be to provide the ability for a Chrome extension to function as a device driver and allow previously new devices to be used – is it cool or what? just think on the ‘old’ days where if you needed to talk with your specific hardware you were locked to write you native application for windows, mac and linux (and more if your users are there). In the new world, you will be able to write it once and run it everywhere… (where have we heard this sentence before? back in the 90s? Some technology that start with J?) One big question is if/when we could see this API being part of the web platform. I don’t really know. However, I do hope it will.

The APIs functions:

  • Locates an instance of the device specified by its vendor and product identifier - chrome.experimental.usb.findDevice(
    integer context,
    integer vendorId,
    integer productId,
    function callback)
  • Performs a USB bulk transfer to the specified device - chrome.experimental.usb.bulkTransfer(integer device,
    string direction,
    integer endpoint,
    string data,
    function callback) 
  • Close a USB device handle - chrome.experimental.usb.closeDevice(integer device,
    undefined callback)
  • Performs a USB control transfer to the specified device - chrome.experimental.usb.controlTransfer(integer device,
    string direction,
    string recipient,
    string type,
    integer request,
    integer value,
    integer index,
    string data,
    function callback)
  • Creates a USB context by which devices may be found - chrome.experimental.usb.createContext(function callback)
  • Disposes of a context that is no longer needed. It is not necessary that this call be made at all, unless you want to explicitly free the resources associated with a context - chrome.experimental.usb.destroyContext(integer context)
  • Performs a USB interrupt transfer to the specified device - chrome.experimental.usb.interruptTransfer(integer device,
    string direction,
    integer endpoint,
    string data,
    function callback)

* This API proposal was on March 7th, 2012. For more details check this proposal. After all we are dealing here with an open source project.

Bluetooth API

A bluetooth API that is on par with the Android and iOS APIs. Version 1 will support basic RFCOMM communication. Profile support will be left for a future version. As for the most common use cases we can think on anything that you are doing today on your mobile device (e.g. headset, stream video/audio etc’). One important aspects to pay attention (just like on mobile devices) will be to see how intensive the bluetooth API is in terms of making your battery drain.

The APIs functions:

  • Accept incoming bluetooth connections by advertising as a service - chrome.experimental.bluetooth.acceptConnections(string uuid,
    string service_name,
    string service_description,
    function callback)
  • Connect to a service on a bluetooth device - chrome.experimental.bluetooth.connect(BluetoothDevice device,
    string uuid,
    function callback)
  • Close the bluetooth connection specified by socket - chrome.experimental.bluetooth.disconnect(BluetoothSocket socket, function callback)
  • Get the bluetooth address of the system - chrome.experimental.bluetooth.getBluetoothAddress(function callback)
  • Request a list of bluetooth devices that support service - chrome.experimental.bluetooth.getDevicesWithService(string service_uuid,
    function callback)
  • Get the local Out of Band Pairing data - chrome.experimental.bluetooth.getOutOfBandPairingData(function callback)
  • Check if this extension has access to bluetooth - chrome.experimental.bluetooth.isBluetoothCapable(function callback)
  • Check if the bluetooth adapter has power - chrome.experimental.bluetooth.isBluetoothPowered(function callback)
  • Read data from a bluetooth connection - chrome.experimental.bluetooth.read(BluetoothSocket socket,
    function callback)
  • Set the Out of Band Pairing data for the bluetooth device at bluetooth_address - chrome.experimental.bluetooth.setOutOfBandPairingData(string bluetooth_address, array of ArrayBuffer data,function callback)
  • Write data to a bluetooth connection - chrome.experimental.bluetooth.write(BluetoothSocket socket, ArrayBuffer data, function callback)
  • Fired when the availability of bluetooth on the system changes - chrome.experimental.bluetooth.onBluetoothAvailabilityChange.addListener(function(boolean available) {...your code...});
  • Fired when the powered state of bluetooth on the system changes - chrome.experimental.bluetooth.onBluetoothPoweredChange.addListener(function(boolean powered) {...your code...});

* This API proposal was on March 7th, 2012. More details can be found in the original proposal.

I know few startups that are waiting for these APIs that they would love to built interesting apps to use them. It’s going to be very interesting to see what new smart-watches, GPSs, Mobile devices etc’ will do with these APIs. Be strong & happy.

Standard
Chrome, webdev

Dart Crawler Example

In the Dart hackathon I got few questions about applications on the server. The best way was to try and give the hackers a code sample… It’s by definition a very simple code but I’m sure that you can take it to the next level without any problem.


#import('dart:io');
#import('dart:uri');
#import('dart:json');
// Dart Hackathon TLV 2012
//
// A simple example to fetch RSS/JSON feed and parse it on the server side
// This is a good start for a crawler that fetch info and parse it.
//
// Author: Ido Green | greenido.wordpress.com
// Date: 28/4/2012
//
class Crawler {
String _urlToFetch = "http://feeds.feedburner.com/html5rocks&quot;;
String _dataFileName = "webPageData.json";
HttpClient _client;
var rssItems;
//Ctor.
Crawler() {
_client = new HttpClient();
}
// Fetch the page and save the data locally
// in a file so we could process it later
fetchWebPage() {
// Get all the updates of h5r
Uri pipeUrl = new Uri.fromString(_urlToFetch);
// open a GET connection to fetch this data
var conn = _client.getUrl(pipeUrl);
conn.onRequest = (HttpClientRequest request) {
request.outputStream.close();
};
conn.onResponse = (HttpClientResponse response) {
print("status code:" + response.statusCode);
var output = new File(_dataFileName).openOutputStream();
response.inputStream.pipe(output);
// In case you want to print the data to your console:
// response.inputStream.pipe(stdout);
};
}
// Read a file and return its content.
readFile() {
File file = new File(_dataFileName);
if (!file.existsSync()) {
print ("Err: Could not find: " + _dataFileName);
return;
}
InputStream file_stream = file.openInputStream();
StringInputStream lines = new StringInputStream(file_stream);
String data = "";
lines.onLine = () {
String line;
while ((line = lines.readLine()) != null) {
//print ("== "+line);
data += line;
}
};
lines.onClosed = () {
print ("Got to the end of: "+_dataFileName);
print ("This is our file content:\n" + data);
parsePage(data);
};
}
//
// Basic (real basic) parsing
//
parsePage(data) {
// cut the intersting part of the feed
int start = data.indexOf("<title>");
int end = data.lastIndexOf("</channel>");
var feed = data.substring(start, end);
// put the items in an array
rssItems = feed.split("<title>");
for (var item in rssItems) {
print("\n** Item: " +item);
}
}
} // End of class
//
// Start the party
//
void main() {
Crawler crawler = new Crawler();
crawler.fetchWebPage();
crawler.readFile();
}

view raw

DartCrawlerClassExample.dart

hosted with ❤ by GitHub

this example could be consider version 0.01 of a real crawler. You do need to add to the real first version features like:

  • Discovery – Be able to get links from the current page and jump into them. This is much harder then it sounds, as you want to make sure it won’t continue forever.
  • Parsing – parse the information on the page. Try to gain the meta data and add it to the ‘real’ content (which is based on your goals from the crawler).
  • Analyze – Meaning, normalize the information of the page and put it in a storage (DB, file, a cloud solution etc’).
  • Logging &Monitoring – As this server side process will run while you are sleeping… It’s best to have some good ‘watch-dog’ on it. The start will be with some simple logging and analyzing of the logs. The second step will be to use a tool to monitor the action.

Key lessons:

  • There is a real need to libraries that will make the parsing better. xPath, DOM to Map (or Array) etc’.
  • The debugging in the editor could improved… and as a first step you might want to use a logging library that will give you a lot of information for each step.
  • The editor making the development phase very nice with warnings on (almost) every issue that you might do. I found it very productive to be back in the good hands of ‘IDE’.
  • I guess that in the near future we will see some good examples that use Dart VM on the server – It’s going to be interesting to profile their performance and see where do we stand vis a vis other modern languages like: Scala.
Standard
Chrome, HTML5, webdev

What Is ChromeOS? In a 5 Minutes Lighting Talk

What is Chrome OS? Well, ChromeOS (and the new Chromebooks) are built and optimized for the web, where users are already spend most of their computing time. Here is a lighting talk I gave in the Java Posse roundup 2012. If you know nothing about Chromebook, ChromeOS and the Chromium Projects – It might be worth your five minutes.  This presentation is built on top of impress.js and you can checkout the code on github.com/greenido/chromeOS-5min

The new chromebooks

You can also checkout the summary of the other talks I gave in that amazing conference.

Standard
Chrome, webdev

Dart Instagram Web App

One (of the many) good things that happend during the Dart hackathon 2012 in Tel Aviv was the ability to hack with friends. There were a lot of interesting project and I had a bit of time to hack this simple web app that show a combination of few tools. The main goal was to investigate and see how we can work with web services in Dart while giving the user a cool UI. First, I’ve looked at how my JavaScript code should look in Dart. Then, it was easy to bake the functionality into the code that fetch images from Instagram. When you have a case where you need to fetch some unstructured data from the web you might want to consider using yahoo pipes (and/or the new version YQL). In our case, I saw that the work on web.stagram is in the area of what I’ll need in terms of data but (like so many other web site) they don’t have any JSON feed I can work with. The option to parse feeds (RSS/Atom) in JavaScript is painful so here y! pipes come to the rescue. This pipe will take the page of ‘photo of the day’ and will give you back a JSON output of all the information you will want to see in a feed from that page. From here the basic code to fetch the JSON and to build the HTML is looking like that:


// init values on the page
  startThePage() {
    String baseurl = "http://pipes.yahoo.com/pipes/pipe.run?_id=8a481ba9ce15f5efa8ac6b894b45eeac&rand=3334&_render=json";
    XMLHttpRequest request = new XMLHttpRequest();
    request.open("GET", baseurl, true);
    request.on.load.add((e) {
      _divCar.hidden = false;
      
      var response = JSON.parse(request.responseText);
      var imgs = response['value']['items'];
      for (final img in imgs) {
        writeCarousel(img);
      }
    });
    request.send();
  }

Other tools/frameworks I’ve used in this mini-project:

  • Dart – I must say that the new language is very easy to pick up. If you are Java programer a lot of things will look (very) familiar (to good and bed). But, even if you spend you last several years hacking on JavaScript – you will feel at home after the first few hours.
  • Twitter bootstrap – These days, it’s one of the best options to have a quality responsive layout with a lot of other CSS goodies.
  • Y! pipes – Instead of taking the time build an RSS to JSON web service (which might be a cool idea for another hackathon) I’ve used pipes that give you not only that but also a fast cache version of the information so you won’t put any load on the servers of your source.
  • The unoffical web.stagram API – In our case, it was the best way to get the ‘photo of the day’ from Instagram.
Overall, it’s very simple code, yet, it’s giving us some views of what can be done with Dart. I would love to put some more time into this project in order to have a nice web app (and not just simple web site as it’s current state).
Standard
Chrome, webdev

Dart Hackathon TLV Summary

Web Workers in the 19th centeryLast Friday and Saturday we hosted a Dart Hackathon in Tel Aviv. When you have a group of people
OK… When hackers, geeks, coders, ninjas and software engineers are coming to spend their weekend hacking on the bleeding edge of technology you know good things will come live. I thought we will have some cool demos in the end but the level of the projects we saw was very impressive. From a generic library of Types to new math game that is doing some clever things with inheritance, canvas and other goodies both on the server side and the client.

Few teams that I would like to mention here:

  • DJ web app – A cool web app that let you ‘play’ the DJ part.
  • Volfied like game, only much better – https://github.com/yanivoliver/DartVolfied
  • GraphMVC – A modular framework for graph (vertices/edges) data structures https://github.com/habeanf/dartgraph
  • Implement the Novem game in dart – It’s a new math game that doesn’t exist on the web (nor on mobile) so they are keeping the source for now. We might have some parts without the algo in Github in the next few days.

All the information about the teams with their ideas and links to their Github repositories

dart

Few hackers came to me during the event with questions about Dart server side. Here is a basic example to server side crawler: https://gist.github.com/2517000 it took me less then ten minutes to write it and I’m sure you can take it from here to the next level. I’ve also had a bit of time (not too much) to work on a web app. It is a simple way to watch cool photos from Instagram on your browser.

The code is in Github under DartInsta and this project uses several technologies:

  • Dart – of course… all the main logic of the web app is written in Dart.
  • Twitter bootstrap – yes, let’s have some good responsive layout without to invest too much effort.
  • Y! pipes for the feeds – who said you can’t enjoy JSON from any web site on the web?
  • Instagram (or the unofficial web.stagram API) – After all, we do need some photos and it’s better to have some real good ones.

Old style dartSome thoughts for the future:

    • Dart is a cool pre alpha technology that (I hope) going to help us build solid web apps without the need to be a ‘guru’. It’s still very (very) early so there are many things that we could improve over time.
    • The community (web developers, Java developers etc’) should try and see what are the libraries the will give the most ‘bang for the buck’. Since it’s so early in the life cycle it will be great to have some libraries that moving everyone forward and not something like the case with jQuery slideshows (= too many not so ‘great’ ones).
    • We should do more events like this but not on weekends so people that can’t drive on Shabbas could join us.
    • Dart is very easy for Java developers. It’s not the case with ‘hard core’ JS ninjas.
    • In case you are going to organize a hackathon here are few great tips.

(*) For hebrew speakers, here is a great explanation that was recorded two days after the event.

Standard