json


What is difference between Post Tool and Index Handlers?


I have a JSON documents which needs to be indexed in Solr. The document looks like this:
{
"id":"1",
"prop":null,
"path":"1.parent",
"_childDocuments_":[
{
"id":"2",
"path":"2.parent.child"
}
]
}
It contains parent-child relationship structure denoted by _childDocuments_ key.
When I insert the documents in Solr via Post Tool, i.e., ./bin/post -c coreName data.json and query the Solr, then I get following response from Solr:
$ curl 'http://localhost:8983/solr/coreName/select?indent=on&q=*:*&wt=json'
{
"responseHeader":{
"status":0,
"QTime":0,
"params":{
"q":"*:*",
"indent":"on",
"wt":"json"}},
"response":{"numFound":1,"start":0,"docs":[
{
"id":"1",
"path":["1.parent"],
"_childDocuments_.id":[2],
"_childDocuments_.path":["2.parent.child"],
"_version_":1566718833663672320}]
}}
But when I try to insert the same JSON document via Index Handler, i.e., curl -
$ curl "http://localhost:8983/solr/coreName/update?commit=true" -H 'Content-type:application/json' --data-binary "#1.json"
{"responseHeader":{"status":400,"QTime":56},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","org.apache.solr.common.SolrException"],"msg":"Unknown command 'id' at [11]","code":400}}
I get SolrException. But if I put the JSON in array, then it shows another error, i.e.,
[{
"id":"1",
"prop": null,
"path":"1.parent",
"_childDocuments_":[
{
"id":"2",
"path":"2.parent.child"
}
]
}]
Error:
$ curl 'http://localhost:8983/solr/coreName/update?commit=true' -H 'Content-type:application/json' --data "#/home/knoldus/practice/solr/1.json"
{"responseHeader":{"status":500,"QTime":3},"error":{"trace":"java.lang.NullPointerException\n\tat org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.mapValueClassesToFieldType(AddSchemaFieldsUpdateProcessorFactory.java:370)\n\tat org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:288)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)\n\tat org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)\n\tat org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)\n\tat org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)\n\tat org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)\n\tat org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)\n\tat org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:118)\n\tat org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)\n\tat org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:91)\n\tat org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:492)\n\tat org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:139)\n\tat org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:115)\n\tat org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:78)\n\tat org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)\n\tat org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:166)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2306)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:534)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tat java.lang.Thread.run(Thread.java:745)\n","code":500}}
So, I have to remove "prop": null as well or make it an empty string, like this:
[{
"id":"1",
"prop": "",
"path":"1.parent",
"_childDocuments_":[
{
"id":"2",
"path":"2.parent.child"
}
]
}]
After making these modifications, when I insert the JSON doc. in Solr via curl, then it works fine.
$ curl 'http://localhost:8983/solr/coreName/update?commit=true' -H 'Content-type:application/json' --data "#/home/knoldus/practice/solr/1.json"
{"responseHeader":{"status":0,"QTime":851}}
And I get following response from Solr query:
$ curl 'http://localhost:8983/solr/coreName/select?indent=on&q=*:*&wt=json'
{
"responseHeader":{
"status":0,
"QTime":1,
"params":{
"q":"*:*",
"indent":"on",
"wt":"json"}},
"response":{"numFound":2,"start":0,"docs":[
{
"id":"2",
"path":["2.parent.child"]},
{
"id":"1",
"path":["1.parent"],
"_version_":1566719240059224064}]
}}
But here again I see a difference that _childDocuments_ have been indexed as separate documents.
So, I have following questions on two different methods of data indexing in Solr:
Why Post Tool ./bin/post does not index _childDocuments_ separately like Request Handler /update?
Why Request Handler /update requires JSON document to be wrapped in array?
And last, why Request Handler /update cannot handle null values whereas Post Tool can?
PostTool is just some small java utility that reads the input you give to it, and sends it to Solr. It is not running inside Solr. I have not looked but I am pretty sure:
it does not detect childDocuments as a special value and sends it just like any other nested json
as explained in the docs, /update requires the array, you can send individual json doc to /update/json/docs
PostTool is converting the null to either "" and just ignoring the field

Related Links

Extract and Create Property in JSON file with RegEx
How can I unmarshall JSONp returned from a request?
Apache Pig error while dumping Json data
Unable to get html string stored in json string variable
asp.net mvc 5 Dapper Json is mapping the whole model class
Here-API load 'NextPageInformation'
Transparently install npm packages offline, from a custom filesystem directory
Parse JSON object in postgresql (Migrating from MongoDB to PSQL)
reading a json file in r lexical error: invalid char in json text
How to specify body in httr POST for json-rpc application content type [R] [closed]
Sending a hash of hashes from a Perl CGI program to another Perl script via a command-line argument
How to extract the Json values in jmeter using regular expression extractor?
How to include `NA` values in .json file
Getting JSON data with VB.Net
Parsing nested arrays in JSON data swift
org.apache.thrift.transport.TTransportException error while Reading large JSON file in zeppelin scala

Categories

HOME
cakephp-3.x
project-management
oauth
json-ld
aem
avro
parse-server
gspread
codeblocks
onenote-api
slim-3
height
unity2d
flexbox
window
gatsby
public-key-encryption
dlib
netezza
scenebuilder
factor-analysis
anychart
pyephem
deb
facebook4j
classpath
devforce
jboss5.x
svnkit
smart-mobile-studio
stocks
fractals
passenger
redhat-datavirt
liquid-xml
elfinder
equivalence
intellij-idea-2016
dojox.mobile
ttcn
context-free-language
archer
rdw
jenkins-jira-trigger
mouseclick-event
livescribe
bnd
modelattribute
hpcc
iphone-developer-program
iron.io
payu
system.web.ui.webcontrols
gradle-script-kotlin
c#-interactive
aescryptoserviceprovider
gameanalytics
nofollow
inotifypropertychanged
coordinate-transformation
mfc-feature-pack
android-viewholder
hornetq
textkit
python-winshell
ipojo
ffprobe
viper-architecture
lines-of-code
ejabberd-saas
libsndfile
grunt-express
emma
internal
gfs
jquery-tabs
bignum
loop-invariant
sslexception
dynamic-binding
entity-framework-4.1
ienumerator
qss
database-permissions
magickwand
htmltextwriter
device-emulation
icsharpcode
getstring
windows-phone-7-emulator
flash-cs5.5
nsindexpath
newsstand-kit
authenticode
scalaxb
ihttphandler
email-spec
msn
unreachable-code
scala-2.8
visual-c++-2008-express
plinq
nt4
self-reference

Resources

Mobile Apps Dev
Database Users
javascript
java
csharp
php
android
MS Developer
developer works
python
ios
c
html
jquery
RDBMS discuss
Cloud Virtualization
Database Dev&Adm
javascript
java
csharp
php
python
android
jquery
ruby
ios
html
Mobile App
Mobile App
Mobile App