feat(phase-1/m2.3): import dedupe — importing the same file twice adds nothing

Phase 1 M2 task 3 (tools/DESKTOP_PHASE_PLAN.md §4). Logic lives in
BackupBridge so both platforms get it through the one bridge:

- BackupBridge.import gains skipExisting (default true) and returns
  ImportSummary(imported, skipped). A note is skipped when the library
  already holds the key (createdAt, updatedAt, sha256(content)) — the
  triple the export preserves verbatim, trustworthy only since the M2
  import-timestamp fix. Keys are snapshotted BEFORE importing, so two
  genuinely identical notes inside one file still both import into a
  fresh library; only a second import of the same file is a no-op.
  Snapshotted keys include trashed notes. skipExisting=false restores
  the old additive behaviour (the documented override).
- Android: BackupManager.import passes the flag and returns the summary
  (signature-identical call for SettingsViewModel); its snackbar now
  reports the skip count. Screen layouts untouched — exit criterion
  "Android backup screens unchanged" holds.
- Desktop: the import confirm dialog swaps its "duplicates persist"
  warning for a working "Skip notes that are already here" checkbox
  (default ON); scripted path prints the re-import outcome.
- Format unchanged: formatVersion stays 1 (streaming/dedupe-at-format
  level stays a Phase-2 formatVersion=2 idea).

Verification passed: :core:data-jvm:backupSmoke (new step — re-import
skipped=27, override added 27, store=54; RESULT M2 round-trip PASS on
the real export) + :desktopApp:run --args="--data-dir /tmp/incido-m2t3
--backup-import <export> --backup-export … --exit-after 8" (imported=27
skipped=0 → exported=27 → reimport imported=0 skipped=27, clean exit) +
assembleDebug + :core:data-jvm:smoke + :desktopApp:run --self-test —
all green; output recorded in DESKTOP_FEASIBILITY.md.

Known limitation (by design): attachments are not in the key.
This commit is contained in:
Incido Developer committed 2026-10-05 19:33:07 +13:00
1 parent cf0e74b68d
commit 031d8570b4
7 files changed
+138 -21

No files matched your search

@@ -62,14 +62,17 @@ class BackupManager(
* [passphrase] is required (caller validates with * [passphrase] is required (caller validates with
* [PasswordPolicy]). * [PasswordPolicy]).
* *
* Returns the number of notes imported. * [skipExisting] (default true) dedupes on the bridge's
* (createdAt, updatedAt, sha256(content)) key — see
* [BackupBridge.import]. Returns what was imported vs skipped.
*/ */
suspend fun import( suspend fun import(
uri: Uri, uri: Uri,
passphrase: CharArray? = null, passphrase: CharArray? = null,
skipExisting: Boolean = true,
onProgress: (current: Int, total: Int) -> Unit = { _, _ -> }, onProgress: (current: Int, total: Int) -> Unit = { _, _ -> },
): Int = withContext(Dispatchers.IO) { ): ImportSummary = withContext(Dispatchers.IO) {
BackupBridge.import(repository, openInput(uri), passphrase, onProgress) BackupBridge.import(repository, openInput(uri), passphrase, skipExisting, onProgress)
} }
/** /**
@@ -243,8 +243,12 @@ class SettingsViewModel(
backupManager.preview(uri, passphrase) backupManager.preview(uri, passphrase)
suspend fun import(uri: android.net.Uri, passphrase: CharArray? = null) { suspend fun import(uri: android.net.Uri, passphrase: CharArray? = null) {
val count = backupManager.import(uri, passphrase) val result = backupManager.import(uri, passphrase)
_snackbar.value = "Imported $count note(s)." _snackbar.value = if (result.skipped > 0) {
"Imported ${result.imported} note(s), skipped ${result.skipped} already imported."
} else {
"Imported ${result.imported} note(s)."
}
} }
/** /**
@@ -6,6 +6,7 @@ import java.io.OutputStream
import java.security.GeneralSecurityException import java.security.GeneralSecurityException
import javax.crypto.AEADBadTagException import javax.crypto.AEADBadTagException
import javax.crypto.BadPaddingException import javax.crypto.BadPaddingException
import kotlinx.coroutines.flow.first
import org.incido.domain.repository.NoteRepository import org.incido.domain.repository.NoteRepository
/** /**
@@ -78,24 +79,45 @@ object BackupBridge {
* encrypted files, [passphrase] is required (caller validates with * encrypted files, [passphrase] is required (caller validates with
* `PasswordPolicy`). * `PasswordPolicy`).
* *
* **Dedupe (M2 task 3):** with [skipExisting] (the default) a note is
* skipped when the library already holds a note with the same
* `(createdAt, updatedAt, sha256(content))` key — the triple the
* export preserves verbatim on both platforms (sound only since the
* M2 import-timestamp fix). The existing keys are snapshotted BEFORE
* this call, so a file containing two genuinely identical notes still
* imports both into a fresh library; only a *second import* of the
* same file adds nothing. Pass `skipExisting = false` to force the
* old additive behaviour.
*
* Errors surface as thrown exceptions, same as the old * Errors surface as thrown exceptions, same as the old
* `BackupManager.import`: missing passphrase → * `BackupManager.import`: missing passphrase →
* [IllegalArgumentException]; bad passphrase / corrupt envelope → * [IllegalArgumentException]; bad passphrase / corrupt envelope →
* `GeneralSecurityException` family. * `GeneralSecurityException` family.
*
* Returns the number of notes imported.
*/ */
suspend fun import( suspend fun import(
repository: NoteRepository, repository: NoteRepository,
input: InputStream, input: InputStream,
passphrase: CharArray? = null, passphrase: CharArray? = null,
skipExisting: Boolean = true,
onProgress: (current: Int, total: Int) -> Unit = { _, _ -> }, onProgress: (current: Int, total: Int) -> Unit = { _, _ -> },
): Int { ): ImportSummary {
val bytes = input.use(InputStream::readBytes) val bytes = input.use(InputStream::readBytes)
val json = payloadJson(bytes, passphrase) val json = payloadJson(bytes, passphrase)
val file = BackupCodec.decode(json) val file = BackupCodec.decode(json)
BackupCodec.importSnapshot(file, repository) { cur, tot -> onProgress(cur, tot) }
return file.notes.size val toImport = if (!skipExisting) {
file.notes
} else {
val existing = repository.observeAllIncludingTrashed().first()
.map { keyFor(it.createdAt, it.updatedAt, it.content) }
.toSet()
file.notes.filterNot { keyFor(it.createdAt, it.updatedAt, it.content) in existing }
}
val skipped = file.notes.size - toImport.size
BackupCodec.importSnapshot(file.copy(notes = toImport), repository) { cur, tot ->
onProgress(cur, tot)
}
return ImportSummary(imported = toImport.size, skipped = skipped)
} }
/** /**
@@ -152,8 +174,26 @@ object BackupBridge {
?: throw IllegalArgumentException("Encrypted backup requires a passphrase") ?: throw IllegalArgumentException("Encrypted backup requires a passphrase")
return EncryptedBackupCodec.decrypt(bytes, pass) return EncryptedBackupCodec.decrypt(bytes, pass)
} }
/**
* M2 task 3 dedupe key: `(createdAt, updatedAt, sha256(content))`.
* Every component survives export/import on both platforms, and the
* timestamps are only trustworthy for dedupe because the repository
* now honours caller-supplied values (M2 import-timestamp fix).
* NUL-separated so distinct field splits can never collide; the
* content rides as a SHA-256 digest to keep the compared set cheap.
*/
private fun keyFor(createdAt: Long, updatedAt: Long, content: String): String {
val digest = java.security.MessageDigest.getInstance("SHA-256")
.digest(content.toByteArray(Charsets.UTF_8))
val hex = digest.joinToString("") { "%02x".format(it) }
return "$createdAt\u0000$updatedAt\u0000$hex"
}
} }
/** Outcome of [BackupBridge.import] — what a duplicate-safe import did. */
data class ImportSummary(val imported: Int, val skipped: Int)
/** Lightweight DTO shown in the import confirmation dialog. */ /** Lightweight DTO shown in the import confirmation dialog. */
data class BackupPreview( data class BackupPreview(
val noteCount: Int, val noteCount: Int,
@@ -74,7 +74,7 @@ fun main(args: Array<String>) {
step("import into fresh JVM library") { step("import into fresh JVM library") {
ObjectBoxStore.init(libDir) ObjectBoxStore.init(libDir)
val n = runBlocking { BackupBridge.import(repo(), FileInputStream(src)) } val n = runBlocking { BackupBridge.import(repo(), FileInputStream(src)).imported }
require(n == original.notes.size) { "imported $n, expected ${original.notes.size}" } require(n == original.notes.size) { "imported $n, expected ${original.notes.size}" }
val inStore = runBlocking { repo().observeAllIncludingTrashed().first().size } val inStore = runBlocking { repo().observeAllIncludingTrashed().first().size }
require(inStore == original.notes.size) { "store holds $inStore" } require(inStore == original.notes.size) { "store holds $inStore" }
@@ -110,7 +110,7 @@ fun main(args: Array<String>) {
ObjectBoxStore.reset() ObjectBoxStore.reset()
val dir2 = File(libDir.absolutePath + "-enc").also { it.deleteRecursively(); it.mkdirs() } val dir2 = File(libDir.absolutePath + "-enc").also { it.deleteRecursively(); it.mkdirs() }
ObjectBoxStore.init(dir2) ObjectBoxStore.init(dir2)
val n = runBlocking { BackupBridge.import(repo(), ByteArrayInputStream(envelope), pw()) } val n = runBlocking { BackupBridge.import(repo(), ByteArrayInputStream(envelope), pw()).imported }
require(n == original.notes.size) { "imported $n, expected ${original.notes.size}" } require(n == original.notes.size) { "imported $n, expected ${original.notes.size}" }
val withAtt = runBlocking { repo().observeAllIncludingTrashed().first() } val withAtt = runBlocking { repo().observeAllIncludingTrashed().first() }
.first { it.attachments.isNotEmpty() } .first { it.attachments.isNotEmpty() }
@@ -121,6 +121,21 @@ fun main(args: Array<String>) {
require(bytes.contentEquals(expected)) { "attachment bytes differ (${bytes.size} vs ${expected.size})" } require(bytes.contentEquals(expected)) { "attachment bytes differ (${bytes.size} vs ${expected.size})" }
"imported $n notes; attachment ${bytes.size}B byte-identical to the phone's" "imported $n notes; attachment ${bytes.size}B byte-identical to the phone's"
} }
step("dedupe: re-import same file") {
ObjectBoxStore.reset()
val dir3 = File(libDir.absolutePath + "-dedupe").also { it.deleteRecursively(); it.mkdirs() }
ObjectBoxStore.init(dir3)
val first = runBlocking { BackupBridge.import(repo(), FileInputStream(src)) }
require(first.imported == original.notes.size && first.skipped == 0) { "first=$first" }
val again = runBlocking { BackupBridge.import(repo(), FileInputStream(src)) }
require(again.imported == 0 && again.skipped == original.notes.size) { "again=$again" }
// The override must still behave like the old additive import.
val forced = runBlocking { BackupBridge.import(repo(), FileInputStream(src), skipExisting = false) }
require(forced.imported == original.notes.size) { "forced=$forced" }
val count = runBlocking { repo().observeAllIncludingTrashed().first().size }
require(count == original.notes.size * 2) { "store holds $count after forced re-import" }
"re-import skipped=${again.skipped}; override added ${forced.imported} (store=$count)"
}
step("close libraries") { step("close libraries") {
ObjectBoxStore.reset() ObjectBoxStore.reset()
"clean" "clean"
@@ -2,9 +2,11 @@ package org.incido.desktop
import androidx.compose.foundation.layout.Arrangement import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Column import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.material.icons.Icons import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.MoreVert import androidx.compose.material.icons.filled.MoreVert
import androidx.compose.material3.AlertDialog import androidx.compose.material3.AlertDialog
import androidx.compose.material3.Checkbox
import androidx.compose.material3.DropdownMenu import androidx.compose.material3.DropdownMenu
import androidx.compose.material3.DropdownMenuItem import androidx.compose.material3.DropdownMenuItem
import androidx.compose.material3.Icon import androidx.compose.material3.Icon
@@ -20,6 +22,7 @@ import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier import androidx.compose.ui.Modifier
import androidx.compose.ui.text.input.PasswordVisualTransformation import androidx.compose.ui.text.input.PasswordVisualTransformation
import androidx.compose.ui.unit.dp import androidx.compose.ui.unit.dp
@@ -66,6 +69,8 @@ fun BackupActions(
var menuOpen by remember { mutableStateOf(false) } var menuOpen by remember { mutableStateOf(false) }
var importStage by remember { mutableStateOf<ImportStage?>(null) } var importStage by remember { mutableStateOf<ImportStage?>(null) }
var exportStage by remember { mutableStateOf<ExportStage?>(null) } var exportStage by remember { mutableStateOf<ExportStage?>(null) }
// M2 task 3: dedupe is ON by default; the confirm dialog exposes it.
var skipDuplicates by remember { mutableStateOf(true) }
fun say(message: String) { scope.launch { snackbar.showSnackbar(message) } } fun say(message: String) { scope.launch { snackbar.showSnackbar(message) } }
@@ -131,6 +136,8 @@ fun BackupActions(
is ImportStage.Confirm -> ConfirmImportDialog( is ImportStage.Confirm -> ConfirmImportDialog(
preview = stage.preview, preview = stage.preview,
skipExisting = skipDuplicates,
onSkipChange = { skipDuplicates = it },
onDismiss = { importStage = null }, onDismiss = { importStage = null },
onConfirm = { onConfirm = {
importStage = null importStage = null
@@ -141,9 +148,15 @@ fun BackupActions(
repository, repository,
FileInputStream(stage.file), FileInputStream(stage.file),
stage.passphrase?.toCharArray(), stage.passphrase?.toCharArray(),
skipExisting = skipDuplicates,
) )
} }
}.onSuccess { say("Imported $it notes") } }.onSuccess {
say(
"Imported ${it.imported} notes" +
if (it.skipped > 0) " · skipped ${it.skipped} already imported" else "",
)
}
.onFailure { say("Import failed: ${it.message}") } .onFailure { say("Import failed: ${it.message}") }
} }
}, },
@@ -225,6 +238,8 @@ private suspend fun startImport(
@Composable @Composable
private fun ConfirmImportDialog( private fun ConfirmImportDialog(
preview: BackupPreview, preview: BackupPreview,
skipExisting: Boolean,
onSkipChange: (Boolean) -> Unit,
onDismiss: () -> Unit, onDismiss: () -> Unit,
onConfirm: () -> Unit, onConfirm: () -> Unit,
) { ) {
@@ -239,12 +254,13 @@ private fun ConfirmImportDialog(
"Exported by Incido ${preview.appVersion} on $exported", "Exported by Incido ${preview.appVersion} on $exported",
style = MaterialTheme.typography.bodySmall, style = MaterialTheme.typography.bodySmall,
) )
Text( Row(verticalAlignment = Alignment.CenterVertically) {
"Existing notes are kept. (Re-importing the same file still duplicates — " + Checkbox(checked = skipExisting, onCheckedChange = onSkipChange)
"dedupe is M2 task 3.)", Text(
style = MaterialTheme.typography.bodySmall, "Skip notes that are already here",
color = MaterialTheme.colorScheme.outline, style = MaterialTheme.typography.bodyMedium,
) )
}
} }
}, },
confirmButton = { TextButton(onClick = onConfirm) { Text("Import") } }, confirmButton = { TextButton(onClick = onConfirm) { Text("Import") } },
@@ -228,13 +228,20 @@ private fun NotesWindow(
"BACKUP preview notes=${preview?.noteCount} " + "BACKUP preview notes=${preview?.noteCount} " +
"attachments=${preview?.attachmentCount} app=${preview?.appVersion}", "attachments=${preview?.attachmentCount} app=${preview?.appVersion}",
) )
val imported = BackupBridge.import(repository, FileInputStream(file)) val first = BackupBridge.import(repository, FileInputStream(file))
println("BACKUP imported=$imported (same calls as the Import… menu)") println(
"BACKUP imported=${first.imported} skipped=${first.skipped} " +
"(same calls as the Import… menu)",
)
if (backupExport != null) { if (backupExport != null) {
val out = File(backupExport) val out = File(backupExport)
val exported = BackupBridge.exportPlain(repository, FileOutputStream(out), DESKTOP_APP_VERSION) val exported = BackupBridge.exportPlain(repository, FileOutputStream(out), DESKTOP_APP_VERSION)
println("BACKUP exported=$exported notes to $out") println("BACKUP exported=$exported notes to $out")
} }
// M2 task 3: importing the same file again must add nothing —
// the dedupe path the confirm dialog's checkbox controls.
val again = BackupBridge.import(repository, FileInputStream(file))
println("BACKUP reimport imported=${again.imported} skipped=${again.skipped} (dedupe ON)")
} }
} }
if (exitAfterSeconds != null) { if (exitAfterSeconds != null) {
+32
View File
@@ -370,5 +370,37 @@ list already has an exit-time AWT/`exitProcess` quirk).
Full suite green after the change: `assembleDebug`, `:core:data-jvm:smoke`, Full suite green after the change: `assembleDebug`, `:core:data-jvm:smoke`,
`backupSmoke`, `:desktopApp:run --self-test`. `backupSmoke`, `:desktopApp:run --self-test`.
**M2 task 3 — import dedupe: DONE.** `BackupBridge.import` now skips notes
whose `(createdAt, updatedAt, sha256(content))` key already exists in the
library (default ON, `skipExisting = false` overrides), and returns
`ImportSummary(imported, skipped)` instead of a bare count. The existing
keys are snapshotted BEFORE the import, so two genuinely identical notes
inside one file still import into a fresh library — only a second import
of the same file adds nothing (matches the §4 wording exactly). Keys
include trashed notes, so restoring around the trash can't duplicate.
`BackupManager.import` (Android shim) passes the flag through and returns
the summary; `SettingsViewModel`'s snackbar gained the skip count (one
line — the screen layouts are untouched). The desktop confirm dialog
replaced its "duplicates persist" warning with a working
"Skip notes that are already here" checkbox (default ON); the export
format did not change (`formatVersion` stays 1).
```
backupSmoke new step: re-import skipped=27; override added 27 (store=54) PASS
:desktopApp:run --backup-import <real export> --backup-export … --exit-after 8
BACKUP imported=27 skipped=0 → exported=27 → reimport imported=0 skipped=27
```
Known limitation (by design, per plan): attachments are not part of the
key — two notes with identical text AND timestamps but different
attachments dedupe as one. Phone side: second import now says
"Imported 0 note(s), skipped 27 already imported." — device smoke still
pending with the rest of task 4.
Full suite green: `backupSmoke` (9 steps incl. dedupe), `assembleDebug`,
`:core:data-jvm:smoke`, scripted desktop run (clean exit),
`:desktopApp:run --self-test`.