Home Data Prep Q&A

Paxata Community Members: Something special in a community experience is coming your way. Stay tuned to this space.
In the meantime, check out the brand new Data Prep for Data Science topic here and the new DataRobot Community.

Visit the official Paxata Documentation portal for all of your doc needs.

Loading multiple datasets in a single project

I need to load multiple datasets in a single project. But the project allows to load only 1 time.

My requirement is to load different files, apply different processing to them and bring them to common format and then row-append them.

How do I do this under a single project in Paxata?

Best Answer

Answers

  • Hello,

    In order to do this, you would need to create Paxata projects for each one of your datasets to bring them to the desired common format. Once this is done in a project, you could publish these projects to the Paxata Library and then create a fresh Paxata project to row append all the desired projects.

    I hope this makes sense to you! Let me know if you need any further clarifications!


  • @Kumar Jayaram
    I don't see any "import" operation as you have shown in screenshot... may be, this is a new feature?

    In my case, the schema of the files itself is very different. I cant append before I do transformations on each single dataset.... 
  • I was able to find the "expand Append option" after which I can see the "import" step. This is nice.
    But what I am missing is that once I reach this step, I am unable to do schema transformation based on Lookups. That step alone is disabled while I am at the import step. All other options are available....Argh Argh! Any solutions
  • Hi @MagmaMan, it is not possible to do add a lookup/append step as a sub-step before a lookup/append. This is a product design decision. You can add the lookup step after the append is performed. Another option is to do your lookup in a separate project, publish the output (answerset), and append the answerset to your main project. Does that help?

  • MagmaManMagmaMan Posts: 15
    edited January 3, 2019 5:18AM
    @mikeblanko
    Thanks..... Ah... I can understand if Append is not allowed again...as it leads to a recursive trap...
    but why not lookup?
    btw.. I already accomplished this using a separate projects.... possibly tag all these projects to make them traceable as well... But, still if all are stored in a single project, it will be really good.
This discussion has been closed.