python


Index JSON files in elasticsearch using Python?


I have a bunch of JSON files(100), which are named as merged_file 1.json, merged_file 2. json and so on.
How do I index all these files into elasticsearch using python(elasticsearch_dsl) ?
I am using this code, but it doesn't seem to work:
from elasticsearch_dsl import Elasticsearch
import json
import os
import sys
es = Elasticsearch()
json_docs =[]
directory = sys.argv[1]
for filename in os.listdir(directory):
if filename.endswith('.json'):
with open(filename,'r') as open_file:
json_docs.append(json.load(open_file))
es.bulk("index_name", "type_name", json_docs)
The JSON looks like this:
{"one":["some data"],"two":["some other data"],"three":["other data"]}
What can I do to make this correct ?
For this task you should be using elasticsearch-py (pip install elasticsearch):
from elasticsearch import Elasticsearch, helpers
import sys, json
es = Elasticsearch()
def load_json(directory):
" Use a generator, no need to load all in memory"
for filename in os.listdir(directory):
if filename.endswith('.json'):
with open(filename,'r') as open_file:
yield json.load(open_file)
helpers.bulk(es, load_json(sys.argv[1]), index='my-index', doc_type='my-type')

Related Links

Python: how can I handle any unhandled exception in an alternative way?
How to roll my own pypi?
Is there a way to check whether function output is assigned to a variable in Python?
Load non-uniform data from a txt file into a msql database
Python scoping problem
How can I strip comments and doc strings from python source code?
How do I display real-time python script output on a website?
how to extract some text by use lxml?
dynamically adding functions to a Python module
Scaling the y-axis with Matplotlib in Python
Shortening a oft-used code segment for testing a return value in Python
What is the best way to internationalize a Python app with multiple i18n domains?
IPv6 decoder for pcapy/impacket
Using paver and nose together with an atypical directory structure
Load an existing many-to-many table relation with sqlalchemy
How do I write this in Ruby/Python? Or, can you translate my LINQ to Ruby/Python?

Categories

HOME
google-apps-script
payment-gateway
tinyos
angular-cli
rotation
bing
windows-7
smartphone
otrs
amazon-product-api
extract
computer-vision
file-io
internet-explorer-11
goutte
xlsx
flann
public-key-encryption
store
jacoco
typeahead
mod-pagespeed
off-canvas-menu
primes
kadanes-algorithm
backpropagation
google-maps-android-api-2
restfb
fileinfo
aurelia-binding
bitcoin-testnet
invoke-command
twitter-bootstrap-2
skygear
pyscripter
oracle-xml-db
geo
data-extraction
npm-shrinkwrap
remote-server
css-counter
lego
ksoap
haskell-pipes
lexical-analysis
sendgrid-templates
distributed-transactions
intersystems-ensemble
amd
z3py
recycle-bin
ogre3d
chrome-remote-desktop
gce
launch
objloader
libtiff.net
sql-server-administration
cjson
manifest.mf
excon
emokit
asynccallback
asp.net-mvc-2
mysqldumpslow
manjaro
digits
google-hadoop
cpan
jcr-sql2
c++-actor-framework
and-operator
wss
phpdocx
voldemort
late-static-binding
jquery-tabs
gamepad
modeshape
argb
adomd.net
dwolla
phpsh
xgettext
mpmovieplayer
webkit-transform
scraperwiki
smooth
squeel
getresource
task-management
jquery-ui-button
nt4
database-diagramming
hardware-infrastructure

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App