30:T742,# _TODO:_ # The database changed. The database has had some big, number of movies increased. The number of datapoints in each movie has increase from 28 to 32 > keys **dropped** from old database are `_id` and `_createdOn` ### tasks The changes to the BMovie database have to be saved by saving the json file and attaching it to the notebook again. Very hackery but something that should not happen often. 1. Update old code to read new database. - This will help with the understanding of the structure and help with adding to the dataset. 2. The adult flag is still false - this can be updated via the IMDB dataset. I have to export a list of all the imdb IDs to get a filtered list of the imdb moves in the bmovie collection - https://observablehq.com/d/bf8f3c571a8d0b8d filter 3. Get a list of all **job** types, **known_for_department** and **department** from crew and cast - other movies. add links to other movies these people have been in. IMDB use multiple databases to reduce database size. BMovies is only 5MB so might not need to, maybe. - - looking at the dataset there are not many that have more than 3 occurrences, so the data abstraction isn't needed 4. Production counties and company data needs to be added for the movies that have missing data. https://observablehq.com/@hellonearthis/b-movies#cell-1373 - this would mean editing an entry in the BMovie dataset. - Missing data could be on wiki-pedia/data references , https://www.allmovie.com/ or https://www.rottentomatoes.com/ - ie: https://en.wikipedia.org/wiki/Death_Bed:_The_Bed_That_Eats - This is compex due to the connection to the themoviedb.org and the structure of production company data (id number for production company), so an alternative could be to make a 2nd database using IMDB data. 5. Make a 2nd database using IMDB data. 31:T47f,Look up API to get definitions: https://developers.themoviedb.org/3/getting-started/daily-file-exports for people http://files.tmdb.org/p/exports/person_ids_MM_DD_YYYY.json.gz 32MB.gzip It didn't include as much data as below. **gender**: _integer_ 0,1,2,3 default: 0 > 0: not stated? 1: Female 2: Male 3: Non-binary # Cast example ```js { adult: false gender: 1 id: 27136 known_for_department: "Acting" name: "Katharine Isabelle" original_name: "Katharine Isabelle" popularity: 20.567 profile_path: "/hX6i25ieoEJEZ1jV81qob2Na0su.jpg" cast_id: 2 character: "Ginger Fitzgerald" credit_id: "52fe453bc3a36847f80c34e1" order: 0 } ``` https://www.themoviedb.org/person/27136-katharine-isabelle https://www.themoviedb.org/t/p/w300_and_h450_bestv2/hX6i25ieoEJEZ1jV81qob2Na0su.jpg # Crew example Includes directors ```js { adult: false gender: 2 id: 43 known_for_department: "Directing" name: "John Fawcett" original_name: "John Fawcett" popularity: 2.754 profile_path: "/ArR5IXmo5yqjH4I6kcO9W5u9rDC.jpg" credit_id: "52fe453bc3a36847f80c34f7" department: "Directing" job: "Director" } ```32:Tc51,# movies that had no production company ### that I have found data for Some many should be labeled as independent as they have not production company? Death Bed: The Bed That Eats: 45795 https://www.imdb.com/title/tt0385639 https://www.themoviedb.org/movie/45795 "George Barry" did it all and couldn't get funding to finish - so he was that production company. https://www.tcm.com/tcmdb/title/560376/death-bed-the-bed-that-eats#articles-reviews?articleId=64935 about the production 1972, some guy named **George Barry** got a camera and some film. So [no production house](https://366weirdmovies.com/213-death-bed-the-bed-that-eats-1977/) https://www.imdb.com/title/tt0385639/companycredits?ref_=ttfc_sa_3 studio/distributor: Cult Epics ? or **George Barry** --- Super Giant: 523864 https://www.imdb.com/title/tt0049809 https://www.themoviedb.org/movie/523864 Production Company: Shintoho Film Distribution Committee https://www.imdb.com/title/tt0049809/companycredits?ref_=tt_dt_co --- Empire of the Dark: 30419 https://www.imdb.com/title/tt0099498 https://www.themoviedb.org/movie/30419 Production Company: The Nautilus Film Company https://www.imdb.com/title/tt0099498/companycredits?ref_=ttfc_sa_3 --- Black Samurai: 18259 https://www.imdb.com/title/tt0074213 https://www.themoviedb.org/movie/18259 Production Company: BJLJ International https://www.imdb.com/search/title/?companies=co0161081 https://en.wikipedia.org/wiki/Black_Samurai#:~:text=Black%20Samurai%20is%20a%201977,and%20screenplay%20credited%20to%20B. --- The Mutilator: 40451 https://www.imdb.com/title/tt0124014 https://www.themoviedb.org/movie/40451 https://en.wikipedia.org/wiki/The_Mutilator Production company: OK Productions --- Llamageddon: 569957 https://www.imdb.com/title/tt4642970 https://www.themoviedb.org/movie/569957 https://www.amazon.com/Llamageddon-Louie-TheLlama/dp/B07KW4NN32 under details https://www.instantwatcher.com/amazon/title/B07KWBZFCB?content_type=1%202&studio=Check%20is%20in%20The%20Mail%20Productions Studio / Production company 'Check is in The Mail Productions' --- Lycan Colony: 550757 https://www.imdb.com/title/tt0795416 https://www.themoviedb.org/movie/550757 Production Company: 'Wits'-End Entertainment' https://www.imdb.com/title/tt0795416/companycredits?ref_=ttloc_sa_4 --- Shark in Venice: 45009 https://www.imdb.com/title/tt1087474 https://www.themoviedb.org/movie/45009 https://www.imdb.com/title/tt1087474/companycredits?ref_=ttfc_sa_3 Production Companies: Millennium Films Venice Sharks --- FDR: American Badass!: 99223 https://www.imdb.com/title/tt1811315 https://www.themoviedb.org/movie/99223 https://www.imdb.com/title/tt1811315/companycredits?ref_=ttfc_sa_3 Production Companies: A Common Thread Street Justice Films --- Behind the Mask: The Rise of Leslie Vernon: 10961 https://www.imdb.com/title/tt0437857 https://www.themoviedb.org/movie/10961 https://www.imdb.com/title/tt0437857/companycredits?ref_=ttfc_sa_3 Production Companies: Glen Echo Entertainment Code Entertainment 33:[{"id":"7e3b4a16c526c3d9260ae081201ffa808b2547f32500bdf2ff1249525b51f468a5526f2c6e67a9856e3f6bd84cdacb73d1336d1d7e52075fa0e3c6aa87456fe5","href":"https://static.observableusercontent.com/files/7e3b4a16c526c3d9260ae081201ffa808b2547f32500bdf2ff1249525b51f468a5526f2c6e67a9856e3f6bd84cdacb73d1336d1d7e52075fa0e3c6aa87456fe5","name":"latest bmovies.txt.json","type":"application/json","lastModified":1673113518696.025,"size":5075701},{"id":"179b72e61fbc76595c2d5b74d59b7e897c4faceebe9edebe94414f22f76c08807f8212098b54096988e14f98dd0549ce74ee51800b008f028cfffaa073d9ecf1","href":"https://static.observableusercontent.com/files/179b72e61fbc76595c2d5b74d59b7e897c4faceebe9edebe94414f22f76c08807f8212098b54096988e14f98dd0549ce74ee51800b008f028cfffaa073d9ecf1","name":"b-movies.json","type":"application/json","lastModified":1652198706138.857,"size":157707},{"id":"10fe160f2d6c541fec0b7a9338269ddf3d3eeed582afd095c6791a0497b154f655e146e0bd974b0aceeb06a962013bad24b727251d4b14f2c50851c3304144fc","href":"https://static.observableusercontent.com/files/10fe160f2d6c541fec0b7a9338269ddf3d3eeed582afd095c6791a0497b154f655e146e0bd974b0aceeb06a962013bad24b727251d4b14f2c50851c3304144fc","name":"noDupsBMovies.json","type":"application/json","lastModified":1674061206726.821,"size":3209650}] 2e:{"title":"B Movies","cells":[{"id":0,"value":"# B Movies\n\n[Sara Vieira, B Movie dataset](https://twitter.com/NikkitaFTW/status/1523700555549274112) https://github.com/SaraVieira/terrible-but-incredible/blob/main/all.json\n\n## The bulk of the data data comes from https://www.themoviedb.org \n\nI have Broken this notebook up into different notebooks that are now part of an [observable collection.](https://observablehq.com/@hellonearthis/b-movies?collection=@hellonearthis/b-movies)\n\nThis notebook is currently used for mangeling the B Movies database.","pinned":false,"hidden":false,"mode":"md"},{"id":1422,"value":"$30","pinned":false,"hidden":false,"mode":"md"},{"id":564,"value":"toc()","pinned":false,"hidden":false,"mode":"ojs"},{"id":82,"value":"// BMovies = d3.json(\"https://legitbackend.wtf/b-movies?limit=200\") // the live dataset #dead","pinned":true,"hidden":false,"mode":"ojs"},{"id":1473,"value":"BMovies = FileAttachment(\"latest bmovies.txt.json\").json() \n// BMovies = FileAttachment(\"b-movies.json\").json() // old DB","pinned":true,"hidden":false,"mode":"ojs"},{"id":1201,"value":"## Finding the duplicates\n\nreturns movie ID's\n\nI only have one duplicate so not going to code the processing of the dups array until i have two.","pinned":false,"hidden":false,"mode":"md"},{"id":936,"value":"// dups = _(BMovies.map((m) => m.id)).countBy().reduce((acc, val, key) => val > 1 ? acc.concat(key) : acc, [])","pinned":true,"hidden":false,"mode":"ojs"},{"id":960,"value":"// What_Dupes = dups.map(d=> BMovies.filter(movie=> movie.id == d))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2240,"value":"// first_dup = _.findIndex(BMovies, ['id', +dups[0]]);","pinned":true,"hidden":false,"mode":"ojs"},{"id":2238,"value":"// last_dup = _.findLastIndex(BMovies, ['id', +dups[0]]);","pinned":true,"hidden":false,"mode":"ojs"},{"id":2232,"value":"// _.isEqual(BMovies[first_dup], BMovies[last_dup]);","pinned":true,"hidden":false,"mode":"ojs"},{"id":2262,"value":"# Remove the duplicate entry","pinned":false,"hidden":false,"mode":"md"},{"id":2624,"value":"bmovies_noDupe = {\n if (remove_dupes>0) {\n\n let noDupsBMovies = []\n let dups = _(BMovies.map((m) => m.id)).countBy().reduce((acc, val, key) => val > 1 ? acc.concat(key) : acc, [])\n let What_Dupes = dups.map(d=> BMovies.filter(movie=> movie.id == d))\n let first_dup = _.findIndex(BMovies, ['id', +dups[0]]);\n let last_dup = _.findLastIndex(BMovies, ['id', +dups[0]]);\n\n if(_.isEqual(BMovies[first_dup], BMovies[last_dup])) {\n noDupsBMovies = BMovies.filter(function(value, index, arr) { return index != last_dup;});\n }\n return noDupsBMovies\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":2622,"value":"viewof remove_dupes = Inputs.button(`remove dupes`)","pinned":false,"hidden":false,"mode":"ojs"},{"id":1420,"value":"\n----\n

\n# Keys scraped from the dataset\n\n${keys.map(k=>`**${k}**, `)}\n

","pinned":false,"hidden":false,"mode":"md"},{"id":2037,"value":"## Key Types:\n\n**adult**: bool \n**backdrop_path**: string \n**belongs_to_collection**: object \n**budget**: number \n**genres**: array \n**homepage**: string \n**id**: number \n**imdb_id**: string \n**original_language**: string \n**original_title**: string \n**overview**: string \n**popularity**: number \n**poster_path**: string \n**production_companies**: array \n**production_countries**: array \n**release_date**: string \n**revenue**: number \n**runtime**: number \n**spoken_languages**: array \n**status**: string \n**tagline**: string \n**title**: string \n**video**: bool \n**vote_average**: number \n**vote_count**: number \n### New Keys added to dataset\n **_images_**: object \n **_videos_**: object \n **_externalids_**: object \n **_releasedates_**: array \n **_cast_**: array \n **_crew_**: array \n\n","pinned":false,"hidden":false,"mode":"md"},{"id":2117,"value":"$31","pinned":false,"hidden":false,"mode":"md"},{"id":2289,"value":"jobs = _.uniq(_.flatten(BMovies.map(m => m.crew.map(c => c.job))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2325,"value":"departments = _.uniq(_.flatten(BMovies.map(m => m.crew.map(c => c.department))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2332,"value":"crew_names = _.uniq(_.flatten(BMovies.map(m => m.crew.map(c => c.name))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2344,"value":"cast_names = _.uniq(_.flatten(BMovies.map(m => m.cast.map(c => c.name))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2348,"value":"cast_known_for_department = _.uniq(_.flatten(BMovies.map(m => m.cast.map(c => c.known_for_department))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2498,"value":"crw = _.flatten(BMovies.map(m => m.crew.map(c => {\n return {name:c.name,id:c.id,known_for_department:c.known_for_department,department:c.department,job:c.job}\n})))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2502,"value":"viewof table = Inputs.table(crw)","pinned":true,"hidden":false,"mode":"ojs"},{"id":2442,"value":"# Cast crew connections\n","pinned":false,"hidden":false,"mode":"md"},{"id":2513,"value":"crew_occurences = _.uniq(_.flatten(BMovies.map(m => m.crew.map(c => c.id))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2507,"value":"crew_occurences2 = _.toPairs( _.pick(_.countBy(crw, 'id'),crew_occurences))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2576,"value":"column 0: person id \ncolumn 1: number of jobs they did","pinned":false,"hidden":false,"mode":"md"},{"id":2527,"value":"viewof table1 = Inputs.table(crew_occurences2)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2534,"value":"cast_occurences = _.uniq(_.flatten(BMovies.map(m => m.cast.map(c => c.id))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2536,"value":"cast_occurences2 = _.toPairs( _.pick(_.countBy(crw, 'id'),cast_occurences))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2541,"value":"viewof table2 = Inputs.table(cast_occurences2)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1995,"value":"

\n\nThere are ${teGenres.length} unique **genres** in the dataset: \n${teGenres.map(k=>`**${k}**, `)}\n

","pinned":false,"hidden":false,"mode":"md"},{"id":1991,"value":"The **imdb_id** id usage is like so. **`https://www.imdb.com/title/tt0210070/`** \nThis can be also found in the **externalids** as it contains for example \n```js\n{id: 9871, \n imdb_id: \"tt0210070\", \n wikidata_id: null, \n facebook_id: null, \n instagram_id: null, \n twitter_id: null}\n```","pinned":false,"hidden":false,"mode":"md"},{"id":2431,"value":"teGenres = _.uniq(_.flatten(BMovies.map(m=> m.genres.map( g => g.name))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":324,"value":"# B Movie Dataset\n\nThere are two tables below, the first is a filtered list and the seconded is raw data.","pinned":false,"hidden":false,"mode":"md"},{"id":14,"value":"viewof search = Inputs.search(BMovies) // search can only find top level data","pinned":true,"hidden":false,"mode":"ojs"},{"id":11,"value":"viewof mo = Inputs.table(search,{\n columns: ['title', \"original_title\", 'popularity', \"budget\", \"revenue\",'id','production_countries','production_companies',\n 'belongs_to_collection','tagline','genres','homepage','imdb_id','original_language','adult'],\n format: {\n production_countries: c => (c.length==0)?'Unknown':c.map(it=> it.name),\n production_companies: c => (c.length==0)?'Unknown':c.map(it=> it.name),\n belongs_to_collection: c=> c.name,\n genres: c=>c.map(it=> it.name)\n },\n // width: {\n // title:80,\n // original_title:80,\n // popularity:63,\n // budget:75,\n // id:50\n // },\n layout:'fixed'\n})","pinned":false,"hidden":false,"mode":"ojs"},{"id":281,"value":"viewof what = Inputs.table(BMovies)","pinned":true,"hidden":false,"mode":"ojs"},{"id":422,"value":"what // view of the data from selected items in above table, ","pinned":true,"hidden":false,"mode":"ojs"},{"id":2382,"value":"mo","pinned":true,"hidden":false,"mode":"ojs"},{"id":1344,"value":"---\n# production companies\n\nThere are ${production_companies.length} production companies for ${BMovies.length} movies. That means there are sometime more that one production company involved in making a movie.\n\nStructure of Production Company\n```\nproduction_companies: Array(2) [\n 0: Object {id: 1950, logo_path: \"/eIoDtHIJKZaZ4hGbKgqC0PzlqJc.png\", name: \"New World Pictures\", origin_country: \"US\"}\n```\n\n[Adding a production company](https://www.themoviedb.org/talk/50e8756c19c2954365140606) is not easy. A workaround might be to have a filtered list of IMDB movies to only include the BMovies.\n\n## No production company listed:\n${md_no_pc.map(m => m)}\n\nAnd if a production company is found then it needs to be added to the movie dataset.","pinned":false,"hidden":false,"mode":"md"},{"id":2179,"value":"# production countries\n\n### No production country listed:\n${md_no_pco.map(m => m)}","pinned":false,"hidden":false,"mode":"md"},{"id":2176,"value":" no_production_country = _.compact(BMovies.map((m,i) => (m.production_countries.length == 0)?i:null))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2218,"value":"md_no_pco = no_production_country.map(m => \n`${BMovies[m].original_title} \nidbm\nmovedb
`\n)","pinned":true,"hidden":false,"mode":"ojs"},{"id":1073,"value":"production_companies = _.uniq(_.flatten(BMovies.map(m=> m.production_companies.map( g => g.name))))","pinned":true,"hidden":false,"mode":"ojs"},{"id":1258,"value":"pc = {\n let f = {}\n for (let i = 0; i < production_companies.length; ++i) {\n \n f[_.snakeCase(production_companies[i])]=[]\n }\n\n BMovies.map(m=> {\n if (m.production_companies.length > 0) {\n // console.log('movie',m.title,'pcs',m.production_companies.length)\n m.production_companies.map( g => {\n // console.log('name',g.name)\n f[_.snakeCase(g.name)].push(m)\n })\n }\n })\n \n return f\n}","pinned":false,"hidden":false,"mode":"ojs"},{"id":2159,"value":" no_production_company = _.compact(BMovies.map((m,i) => (m.production_companies.length == 0)?i:null))","pinned":true,"hidden":false,"mode":"ojs"},{"id":2184,"value":"md_no_pc = no_production_company.map(m => \n`${BMovies[m].original_title} \nidbm\nmovedb
`\n)","pinned":false,"hidden":false,"mode":"ojs"},{"id":2354,"value":"# Edit bmovie data\n\nThe two entry points I want to edit are **production_companies** and **production_countries**. \n\nThere is sometimes more than one production company\n\nI could just push the production company name onto `BMovies[select].production_companies`\n\n\nfiles to edit:\n\n `no_production_company = Array(9) [14, 25, 40, 57, 107, 115, 120, 128, 159]`\n\n `no_production_country = Array(5) [40, 120, 128, 159, 172]`","pinned":false,"hidden":false,"mode":"md"},{"id":2376,"value":"BMovies[select].original_title ","pinned":true,"hidden":false,"mode":"ojs"},{"id":2379,"value":"BMovies[select].production_companies","pinned":true,"hidden":false,"mode":"ojs"},{"id":2367,"value":"BMovies[select]","pinned":true,"hidden":false,"mode":"ojs"},{"id":2360,"value":"viewof select = Inputs.select(no_production_company, {label: \"Films with no listed production company.\"})","pinned":false,"hidden":false,"mode":"ojs"},{"id":2554,"value":"# add production company","pinned":false,"hidden":false,"mode":"md"},{"id":2557,"value":"viewof pc_text = Inputs.text({label: \"Type in Production Complany name.\"})","pinned":true,"hidden":false,"mode":"ojs"},{"id":2559,"value":"viewof set_text = Inputs.button(`Set '${pc_text}'' to: ${BMovies[select].original_title} `)","pinned":true,"hidden":false,"mode":"ojs"},{"id":2567,"value":"{\n if (set_text>0) {\n BMovies[select].production_companies.push(pc_text)\n return BMovies[select].original_title\n }\n}","pinned":true,"hidden":false,"mode":"ojs"},{"id":1373,"value":"$32","pinned":false,"hidden":false,"mode":"md"},{"id":1075,"value":"# Keys used by tmdb","pinned":false,"hidden":false,"mode":"md"},{"id":24,"value":"keys = _.keysIn(BMovies[0])","pinned":true,"hidden":false,"mode":"ojs"},{"id":2018,"value":"---","pinned":false,"hidden":false,"mode":"md"},{"id":1338,"value":"# IMDB path\n\nIMDB path https://www.imdb.com/title/\n\nmovie **id** : [https://www.themoviedb.org/movie/**849749**](https://www.themoviedb.org/movie/849749)","pinned":false,"hidden":false,"mode":"md"},{"id":562,"value":"import {toc} from \"@nebrius/indented-toc\"","pinned":true,"hidden":false,"mode":"ojs"},{"id":2429,"value":"Style to make the text area wider in observable","pinned":false,"hidden":false,"mode":"md"},{"id":489,"value":"","pinned":false,"hidden":false,"mode":"html"}],"files":"$W33","stdlib":"1","titling":"auto"}