blance/melissa-uploader
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace hand-edited JSON with a setup wizard

Browse Source 
Melissa had to author entity.json by hand and know FTPS from SFTP to do
it - which is what broke the last two CAMBS runs. setup-entity.bat now
asks the questions and writes the file:

- protocol is a numbered menu, stored as an explicit "protocol" key that
  wins over the sftp:// host prefix and the older tls flag
- pasted schemes and paths are stripped off the host before saving
- it creates the practice folders, then logs in and checks every
  practice's remote folder exists, so a wrong path or protocol surfaces
  in seconds instead of in the next morning's run

Also drops insurance_file_type from the required path: with no setting
the engine sends every file in the date folder minus Windows clutter, so
"No .pdf files" can't happen on a folder full of TIFs. Connection
failures now print what to change rather than just WinSCP's wording.
This commit is contained in:
blance
2026-08-26 20:03:00 -05:00
parent c5af810806
commit 0b6f8cb16d
6 changed files with 890 additions and 224 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+143 -185
View File
@@ -1,104 +1,157 @@
# Melissa Uploader
Automated upload tool for practice exports. Uploads insurance PDFs and patient files to the correct FTP/SFTP servers for each practice, then archives what was sent.
Uploads each practice's daily files to the right server, then archives what was
sent. There are three buttons and no files to edit by hand.
| Double-click this | When |
|---|---|
| **setup.bat** | Once, on a new PC. Downloads the uploader's helper program. |
| **setup-entity.bat** | Whenever a partner is added or a password/folder changes. Asks questions and writes the settings for you. |
| **run-all.bat** | Every day, after the PT STMT rename script. |
---
## First-time setup
## Setting up a partner
1. Double-click **setup.bat** — this downloads WinSCP (needed for uploading)
2. Set up your entity folders (see below)
3. Double-click **run-all.bat** each day after running the PT STMT rename script
Double-click **setup-entity.bat** and pick "Set up a new entity". It asks:
- **Entity name** - the folder name for this group of practices (CAMBS, RMI, ...)
- **Server address** - just the name, like `sft.polluxsystems.com`
- **How the server receives files** - pick SFTP, FTPS, or FTP from a list.
If you don't know, ask the partner which one they use. Guessing wrong is the
single most common reason an upload fails, and the test at the end catches it.
- **Username and password**
- **Where the files go on the server** - type `{practice}` where the practice
name belongs and it gets filled in for each one, so a single line covers
every practice:
```
/home/clients/CAMBS/{practice}
```
For CAMBS that sends AJMATS's files to `/home/clients/CAMBS/AJMATS`,
CIIR's to `/home/clients/CAMBS/CIIR`, and so on.
- **Whether PT STMT files show up in the dated folders**, and where those go
- **Whether this entity sends patient files** from `Patient\Export\`
- **The practice names** - it creates each practice folder for you
Then it offers to test. Say yes.
---
## Daily workflow
## The test is the important part
1. Run the **PT STMT rename script** manually (as usual)
2. Double-click **run-all.bat** to upload everything
3. Check the window for any failures — failed files stay in place for retry
At the end of setup (or any time, via "Test an entity's connection and
folders") it logs in and checks that every practice's folder actually exists
on the server:
```
Daily upload
Connecting to sft.polluxsystems.com over SFTP as cambs-user ... OK
/home/clients/CAMBS/AJMATS ... found
/home/clients/CAMBS/CIIR ... found
/home/clients/CAMBS/EMMD ... NOT THERE
```
**OK** means the address, protocol, and login are all right.
**found** means files sent there will land.
**NOT THERE** means that practice will fail - either the path is wrong (fix it
in setup) or the partner hasn't created the folder yet (ask them).
Running this after setup means you find out about a problem now, in ten
seconds, instead of in tomorrow's upload.
---
## Daily run
1. Run the **PT STMT rename script** as usual
2. Double-click **run-all.bat**
3. Watch for red lines
Every file that uploads prints `ok`. A date folder only gets archived when
**everything** in it went up - if anything fails or gets skipped, the folder
stays put so tomorrow's run picks it up again. Nothing is ever deleted.
You do not have to tell it whether the files are PDFs or TIFs. It sends
whatever the imaging system put in the dated folder.
---
## When something goes wrong
The run tells you what to do in plain words. The common ones:
| What you see | What to do |
|---|---|
| `This server wants SFTP...` | Run setup-entity.bat, change that entity's server, pick the protocol it names |
| `The username or password was rejected` | Run setup-entity.bat and retype the login |
| `That server address could not be found` | Typo in the address - fix it in setup |
| `No answer from the server` | Ask IT whether the firewall allows it |
| `... not configured (...) - N file(s) ignored, left in place` | That upload has no folder path set yet. Run setup-entity.bat and fill it in, or ignore the message if that entity genuinely doesn't send those files |
| `NOT archived - N file(s) were ignored` | Some files went up and some didn't, so the folder was left alone on purpose. Fix the setting above and run again |
| `WinSCP error on startup` | Run setup.bat again |
Full detail for every run is in `Logs\YYYYMMDD.log`.
---
## Folder layout
```
run-all.bat <- double-click to run everything
run-all.bat <- daily
setup-entity.bat <- add or change a partner
setup.bat <- once per PC
engine.ps1
setup.bat
WinSCP\ <- created by setup.bat automatically
WinSCP\ <- created by setup.bat
Logs\ <- one log per day
Entities\
CAMBS\
entity.json <- server details for this entity
run.bat <- double-click to run just CAMBS
entity.json <- written by setup-entity.bat, don't edit by hand
run.bat <- runs just CAMBS
Export1\
AJMAT\
practice.json <- only needed if AJMAT has its own FTP login
20260629\ <- today's insurance PDFs go here
Archive sent to FTP\ <- uploaded date folders move here
AJMATS\
20260826\ <- today's files, from the imaging system
Archive sent to FTP\ <- where they go once sent
Patient\
Export\
PT STMT_20260629\ <- created by rename script
Archive Sent to FTP\ <- uploaded patient folders move here
PT STMT_20260826\ <- from the rename script
Archive Sent to FTP\
CIIR\
...
CONSensio\
entity.json
...
```
---
## The two config files
---
There are two JSON files, and they work together — one sets the defaults for
the whole entity, the other overrides just what's different for one practice.
# Reference
- **`entity.json`** — lives in the entity folder (e.g. `Entities\CAMBS\entity.json`).
Required. This is the default server info used by every practice under that
entity: which server, which login, which folder paths.
- **`practice.json`** — lives inside one practice's folder (e.g.
`Entities\CAMBS\Export1\AJMAT\practice.json`). Optional — only add it if a
practice needs something *different* from `entity.json`. Whatever fields you
put in here replace the matching field from `entity.json` for that practice
only; anything you leave out still comes from `entity.json`.
Everything below is for whoever maintains the tool. **setup-entity.bat writes
all of it** - you should never need this to use the uploader.
Common reasons to add a `practice.json`:
- The practice has its own separate FTP/SFTP login instead of the shared one
- The practice's files need to go to a **different folder path** on the
server than the standard `{practice}` pattern (e.g. the imaging vendor set
it up with a different folder name, or a nonstandard structure) — override
`tiff_path`, `pdf_path`, and/or `patient_path` to whatever that
practice actually needs
## entity.json
You can mix and match — e.g. override just `pdf_path` for one practice and
leave its login and every other path alone.
### `entity.json`
1. Create a folder under `Entities\` with the entity name (e.g. `Entities\CAMBS\`)
2. Copy `run.bat` into it
3. Create `entity.json` in that folder — copy from `Entities\EXAMPLE\entity.json`
and fill in your real values. A complete file looks like this:
Written into each entity folder. `setup-entity.bat` produces:
```json
{
"_note": "Written by setup-entity.bat ...",
"workflow": "insurance+patient",
"insurance_file_type": "pdf",
"ftps": {
"host": "ftp.example.com",
"port": 21,
"tls": true,
"host": "sft.polluxsystems.com",
"protocol": "sftp",
"port": null,
"username": "shared-username",
"password": "shared-password",
"tiff_path": "/{practice}/Insurance",
"pdf_path": "/{practice}/PDF"
"tiff_path": "/home/clients/CAMBS/{practice}",
"pdf_path": null
},
"sftp": {
"host": "sftp.example.com",
"port": 22,
"protocol": "sftp",
"port": null,
"username": "shared-username",
"password": "shared-password",
"patient_path": "/{practice}/Patient"
@@ -106,149 +159,54 @@ leave its login and every other path alone.
}
```
4. Create the `Export1\` folder and a subfolder for each practice inside the entity folder
The section names are historical: **`ftps` is the daily date-folder upload**
(over whatever `protocol` says, SFTP included) and **`sftp` is the patient-file
upload**.
---
## Every setting, explained
All of these can appear in `entity.json` (entity-wide default) — and everything
except `workflow` and `insurance_file_type` can also appear in a
`practice.json` to override or add a value for just that practice.
### Top level (entity.json only)
### Top level
| Setting | Values | What it does |
|---------|--------|--------------|
| `workflow` | `"insurance"` or `"insurance+patient"` | `insurance` = only the daily date-folder upload. `insurance+patient` = also look for `Patient\Export\PT STMT_<date>\` folders and send their `.txt`/`.tif` files via SFTP |
| `insurance_file_type` | `"pdf"` (default if null) or `"tif"` | Which file extension to pick up from the daily date folder. TIF entities: CAMBS, RMI, CONSENSIO, INLAND. The PT STMT split works either way (it goes by file *name*, not type) |
|---|---|---|
| `workflow` | `"insurance"` or `"insurance+patient"` | `insurance+patient` also processes `Patient\Export\PT STMT_<date>\` via the `sftp` section |
| `insurance_file_type` | `"pdf"`, `"tif"`, or absent | Optional. Absent (the normal case) sends every file in the date folder except `Thumbs.db`, `desktop.ini`, `.DS_Store`, and `.tmp/.lnk/.ini/.db`. Set it to pin the upload to one extension |
### `ftps` section — the daily date-folder upload
### `ftps` (daily upload) and `sftp` (patient files)
| Setting | Values | What it does |
|---------|--------|--------------|
| `host` | server name or URL | Where the daily files go. Three forms: plain name (`ftp.example.com`) = FTP or FTPS depending on `tls`; `sftp://server.com` = send over **SFTP** instead (some partners receive everything by SFTP — Pollux is one); `ftps://server.com` = force secure FTP regardless of `tls` |
| `port` | number, no quotes | Only needed if nonstandard. Null = 21 for FTP/FTPS, 22 for `sftp://` hosts |
| `tls` | `true` / `false`, no quotes | `true` = secure FTPS, `false`/null = plain FTP. Ignored when `host` starts with `sftp://` (SFTP is always encrypted) |
| `username` | text | Login for this server |
| `password` | text | Password for this server |
| `tiff_path` | remote folder path | Where the regular daily files (insurance TIFs/PDFs) land on the server, e.g. `"/home/clients/CAMBS/{practice}"`. The older name `insurance_path` still works |
| `pdf_path` | remote folder path | Where PT STMT-named files from the date folder land. Null = this entity has no PT STMTs; if one shows up anyway it's left in place with a notice |
|---|---|---|
| `host` | server name | Bare hostname. A pasted `sftp://` / `ftps://` prefix is still honored and stripped |
| `protocol` | `"sftp"`, `"ftps"`, `"ftp"` | How to connect. Wins over any prefix on `host` and over `tls`. Null falls back to the `host` prefix, then `tls`, then FTP for the daily section / SFTP for the patient section |
| `port` | number, no quotes | Null = 22 for SFTP, 21 for FTP/FTPS |
| `tls` | `true`/`false` | Older equivalent of `protocol: "ftps"`. Only read when `protocol` is null |
| `username` / `password` | text | The login |
| `tiff_path` | path | `ftps` only. Where the regular daily files go. The older name `insurance_path` still works |
| `pdf_path` | path | `ftps` only. Where PT STMT-named files *from the date folder* go. Null = ignore them |
| `patient_path` | path | `sftp` only. Where the `.txt`/`.tif` patient files go |
### `sftp` section — the patient-files upload (only used when `workflow` is `insurance+patient`)
| Setting | Values | What it does |
|---------|--------|--------------|
| `host` | server name or URL | Patient-files SFTP server (a `sftp://` prefix is fine and stripped automatically) |
| `port` | number, no quotes | Null = 22 |
| `username` | text | Login |
| `password` | text | Password |
| `patient_path` | remote folder path | Where the `.txt`/`.tif` patient files land, e.g. `"/upload/{practice}/Patient"` |
### Special values that work inside any path
| Token | Becomes |
|-------|---------|
| `{practice}` | The practice's local folder name (e.g. `AJMATS`, `CIIR`) — lets one path line serve every practice under the entity |
### Special values that work anywhere
### Values that work anywhere
| Value | Meaning |
|-------|---------|
| `null` (no quotes) | "We don't use this — ignore it." Files that would need the setting are left in place with a yellow notice, never uploaded, never archived, never counted as failures |
| any key starting with `_` (like `"_comment"`) | Ignored by the tool — use it for notes to yourself inside the file |
|---|---|
| `{practice}` in a path | Replaced with the practice's folder name |
| `null` | "Not used - ignore." Affected files are left in place with a notice, never uploaded, never archived, never counted as failures |
| any key starting with `_` | Ignored - use it for notes |
**Missing keys fix themselves:** on every run, the tool checks `entity.json`
for any keys it knows about that aren't in the file, adds them with a value
of `null`, and saves the file — so you never have to type key names by hand.
A `null` value means **"we don't use this — ignore it"**: any files that
would need that setting are left in place untouched (with a yellow notice in
the output), not treated as failures. To turn a feature on, just open
`entity.json` and replace the `null` with a real value in quotes.
On every run the engine adds any keys it knows about that are missing from
`entity.json`, as `null`, and saves the file.
**Editing JSON — the rules that bite:**
- Key names must match **exactly** (all lowercase, underscores): `host`, not
`hostname`; `username`, not `user`; `tiff_path`, not `tiffpath`
- Every `"key": "value"` pair ends with a comma **except the last one in its
block** — a missing or extra comma breaks the whole file
- Keep values inside straight double quotes (`"`). Avoid editing in Word or
anything that turns quotes curly — use Notepad
- `port` numbers and `tls` true/false do **not** get quotes; everything else does
## practice.json
---
## `practice.json` — overriding one practice
If one practice needs a different login, a different folder path, or both:
1. Create a `practice.json` file inside that practice's folder (next to its
date folders, e.g. `Entities\CAMBS\Export1\AJMAT\practice.json`)
2. Copy from `Entities\EXAMPLE\Export1\PRACTICE_NAME\practice.json`
3. Delete every line you *don't* need to change — keep only the fields that
differ for this practice. Everything you delete falls back to `entity.json`.
The field names are the same ones as `entity.json`, wrapped in the same
`ftps` / `sftp` sections. A `practice.json` can also **add** a field that
`entity.json` doesn't have at all — e.g. if `entity.json` has no
`tiff_path` because every practice's path is different, each practice's
`practice.json` can carry its own.
Example — this practice only has its own PDF folder name, nothing else differs:
Optional, inside one practice's folder. Same keys and sections; whatever it
contains replaces (or adds to) `entity.json` for that practice only. Use it
when one practice has its own login or a remote path that doesn't fit the
`{practice}` pattern.
```json
{
"ftps": {
"pdf_path": "/{practice}/PDF_Statements_Custom"
"tiff_path": "/SpecialFolder/Claims"
}
}
```
Example — this practice has its own SFTP login but uses the same paths as everyone else:
```json
{
"sftp": {
"username": "ajmat-sftp-user",
"password": "ajmat-sftp-pass"
}
}
```
Example — this practice has completely different remote folders for everything:
```json
{
"ftps": {
"tiff_path": "/SpecialFolder/Claims",
"pdf_path": "/SpecialFolder/Statements"
},
"sftp": {
"patient_path": "/SpecialFolder/Patients"
}
}
```
Note: a `practice.json` never needs `workflow` or `insurance_file_type` —
those are entity-wide only.
---
## Something went wrong?
- **Files failed to upload** — they stay in the date folder. Fix the issue and run again.
- **A date folder is still there after running** — one or more files failed. Check the output window.
- **WinSCP error on startup** — run `setup.bat` again to re-download WinSCP.
- **`... upload not configured (ftps.xyz is null/blank) - N file(s) ignored`** —
there are files to send, but the listed settings are still `null` (or blank)
in that entity's `entity.json`. If those files *should* upload, open
`entity.json` and fill in real values for the listed keys. If that upload
type genuinely doesn't apply to this entity, the message is normal and the
files just stay where they are.
- **`Added missing key(s) to entity.json ...`** — informational, not an error.
The tool added keys that were absent (as `null`) so they're ready to fill in.
- **`No .pdf files in <date>\` but the folder isn't empty** — the files in the
date folder are probably TIFs. Set `"insurance_file_type": "tif",` at the top
level of that entity's `entity.json` (right under `workflow`).
- **The startup banner shows the wrong file type** — the banner line
(`Entity: ... | File type: .pdf | ...`) shows what was actually read from
`entity.json`, so it's the quickest way to confirm your edit took effect.
`workflow` and `insurance_file_type` are entity-wide and ignored here.