Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cablecar - Run your own search engine. Today.

Cablecar - Run your own search engine. Today.

An overview about the project and tutorial for getting started with Cablecar and elasticsearch. Cablecar is a node.js frontend for elasticsearch with attachments plugin.

Robert Kowalski

January 31, 2013
Tweet

More Decks by Robert Kowalski

Other Decks in Programming

Transcript

  1. Why not use the ones that exist? What if they

    - or their countries - don't like our data? What if we want to see other things than the average customer? What if we don't want our main search engine to save our queries?
  2. Why not use the ones that exist? Or what if

    I want to create my own search engine - just for me, with my data, at home... OpenSource here I come!
  3. node.js "C10k problem" no Threads - a lot of concurrent

    requests possible "non-blocking I/O" modele
  4. elasticsearch Search Engine based on Apache Lucene adds: REST API,

    easy Clustering... related Projects: Apache Solr
  5. Creating a mapping for the files curl -X PUT "localhost:9200/books/attachment/_mapping"

    -d '{ "attachment": { "properties": { "file": { "type": "attachment", "fields": { "title": {"store": "yes"}, "download": {"store": "yes"}, "filename": {"store": "yes"}, "file": {"term_vector": "with_positions_offsets", "store":" yes"} } } } } }'
  6. #/bin/sh - Indexing one file #!/bin/sh file=RedisManual.pdf encoded=`cat $file |

    perl -MMIME::Base64 -ne 'print encode_base64($_)'` json="{\"file\": \"${encoded}\", \"filename\": \"${file}\"}" echo "$json" > json.file curl -X POST "localhost:9200/books/attachment" -d @json.file
  7. Example - Search curl -X POST http://127.0.0.1:9200/_search?pretty=true \ -d '{"fields":

    ["title", "filename", "download"], "query" { "query_string": { "query": "Data" }}, "highlight": {"fields": {"file": {}}}}'
  8. Example - Result { "took" : 112, "timed_out" : false,

    "_shards" : { "total" : 11, "successful" : 11, "failed" : 0 }, "hits" : { "total" : 11, "max_score" : 0.062009797, "hits" : [ { "_index" : "books", "_type" : "attachment", "_id" : "HbY-6si5QxCA53DqCreZow", "_score" : 0.062009797, "fields" : { "filename" : "RedisManual.pdf", "download" : "http://myserver.com/download/pdf/" }], "highlight" : { "file" : [ "Environment for <em>Data</em> Analysis and Graphics\n\nVersion 2.15.1 (2012-06- 22)\n\nW. N. Venables, D. M. Smith\nand the R", "6\n1.11 <em>Data</em> permanency and removing objects . . 6\n\n2 Simple", "subsets of a <em>data</em> [...]" ] } }}}
  9. The Future of the Project: Extensions Feed the backend with

    a webspider As a separate service no auto-indexing, index on request follow robot.txt? Inverse? not at all? can handle username / password combination