This study will be a 2-part Phase I study comprising a dose-finding escalation part (to determine MTD/RP2D, safety and PK) followed by an extension part at RP2D. In the dose-escalation part, successive cohorts of 3 patients will receive Pexidartinib (given orally every day at escalating doses, five dose levels possible, Plexxikon), in combination with Durvalumab (given IV every 4 weeks at a fixed dose of 1500mg, AstraZeneca). The dose escalation scheme will be done using a Likelihood Continual Reassessment Method (CRML). After each new cohort of 3 patients is evaluated, model will be fitted and results will be discussed in a teleconference between the investigators, the sponsor, representatives of AstraZeneca and Plexxikon, and the statistician who will decide of the next dose to be assigned to the next cohort (Dose Escalation Meetings). In both parts, patients will continue to receive study drugs as long as they experience clinical benefit in the opinion of the investigator or until unacceptable toxicity including DLTs or symptomatic deterioration attributed to disease progression as determined by the investigator after an integrated assessment of radiographic data and clinical status or withdrawal of consent. If a DLT or a toxicity meeting the DLT definition but occurring outside of the DLT period, resolves to Grade ≤2 or the patient's baseline level within 14 days after toxicity onset, dosing may be resumed if agreed by the Sponsor at the same DL. The extension part will comprise two independent cohorts of patients with pancreatic adenocarcinoma (PDAC cohort) or colorectal adenocarcinoma (CRC cohort) and biopsiable disease, and will allow the assessment of anti-tumor activity based on a Gehan design allowing to quickly identify treatments with low efficacy.