feat(phase-1/m2.3): import dedupe — importing the same file twice adds nothing

Phase 1 M2 task 3 (tools/DESKTOP_PHASE_PLAN.md §4). Logic lives in
BackupBridge so both platforms get it through the one bridge:

- BackupBridge.import gains skipExisting (default true) and returns
  ImportSummary(imported, skipped). A note is skipped when the library
  already holds the key (createdAt, updatedAt, sha256(content)) — the
  triple the export preserves verbatim, trustworthy only since the M2
  import-timestamp fix. Keys are snapshotted BEFORE importing, so two
  genuinely identical notes inside one file still both import into a
  fresh library; only a second import of the same file is a no-op.
  Snapshotted keys include trashed notes. skipExisting=false restores
  the old additive behaviour (the documented override).
- Android: BackupManager.import passes the flag and returns the summary
  (signature-identical call for SettingsViewModel); its snackbar now
  reports the skip count. Screen layouts untouched — exit criterion
  "Android backup screens unchanged" holds.
- Desktop: the import confirm dialog swaps its "duplicates persist"
  warning for a working "Skip notes that are already here" checkbox
  (default ON); scripted path prints the re-import outcome.
- Format unchanged: formatVersion stays 1 (streaming/dedupe-at-format
  level stays a Phase-2 formatVersion=2 idea).

Verification passed: :core:data-jvm:backupSmoke (new step — re-import
skipped=27, override added 27, store=54; RESULT M2 round-trip PASS on
the real export) + :desktopApp:run --args="--data-dir /tmp/incido-m2t3
--backup-import <export> --backup-export … --exit-after 8" (imported=27
skipped=0 → exported=27 → reimport imported=0 skipped=27, clean exit) +
assembleDebug + :core:data-jvm:smoke + :desktopApp:run --self-test —
all green; output recorded in DESKTOP_FEASIBILITY.md.

Known limitation (by design): attachments are not in the key.
This commit is contained in:
Incido Developer committed 2026-10-05 19:33:07 +13:00
1 parent cf0e74b68d
commit 031d8570b4
7 files changed
+136 -19

No files matched your search

@@ -62,14 +62,17 @@ class BackupManager(
* [passphrase] is required (caller validates with
* [PasswordPolicy]).
*
* Returns the number of notes imported.
* [skipExisting] (default true) dedupes on the bridge's
* (createdAt, updatedAt, sha256(content)) key — see
* [BackupBridge.import]. Returns what was imported vs skipped.
*/
suspend fun import(
uri: Uri,
passphrase: CharArray? = null,
skipExisting: Boolean = true,
onProgress: (current: Int, total: Int) -> Unit = { _, _ -> },
): Int = withContext(Dispatchers.IO) {
BackupBridge.import(repository, openInput(uri), passphrase, onProgress)
): ImportSummary = withContext(Dispatchers.IO) {
BackupBridge.import(repository, openInput(uri), passphrase, skipExisting, onProgress)
}
/**
@@ -243,8 +243,12 @@ class SettingsViewModel(
backupManager.preview(uri, passphrase)
suspend fun import(uri: android.net.Uri, passphrase: CharArray? = null) {
val count = backupManager.import(uri, passphrase)
_snackbar.value = "Imported $count note(s)."
val result = backupManager.import(uri, passphrase)
_snackbar.value = if (result.skipped > 0) {
"Imported ${result.imported} note(s), skipped ${result.skipped} already imported."
} else {
"Imported ${result.imported} note(s)."
}
}
/**
@@ -6,6 +6,7 @@ import java.io.OutputStream
import java.security.GeneralSecurityException
import javax.crypto.AEADBadTagException
import javax.crypto.BadPaddingException
import kotlinx.coroutines.flow.first
import org.incido.domain.repository.NoteRepository
/**
@@ -78,24 +79,45 @@ object BackupBridge {
* encrypted files, [passphrase] is required (caller validates with
* `PasswordPolicy`).
*
* **Dedupe (M2 task 3):** with [skipExisting] (the default) a note is
* skipped when the library already holds a note with the same
* `(createdAt, updatedAt, sha256(content))` key — the triple the
* export preserves verbatim on both platforms (sound only since the
* M2 import-timestamp fix). The existing keys are snapshotted BEFORE
* this call, so a file containing two genuinely identical notes still
* imports both into a fresh library; only a *second import* of the
* same file adds nothing. Pass `skipExisting = false` to force the
* old additive behaviour.
*
* Errors surface as thrown exceptions, same as the old
* `BackupManager.import`: missing passphrase →
* [IllegalArgumentException]; bad passphrase / corrupt envelope →
* `GeneralSecurityException` family.
*
* Returns the number of notes imported.
*/
suspend fun import(
repository: NoteRepository,
input: InputStream,
passphrase: CharArray? = null,
skipExisting: Boolean = true,
onProgress: (current: Int, total: Int) -> Unit = { _, _ -> },
): Int {
): ImportSummary {
val bytes = input.use(InputStream::readBytes)
val json = payloadJson(bytes, passphrase)
val file = BackupCodec.decode(json)
BackupCodec.importSnapshot(file, repository) { cur, tot -> onProgress(cur, tot) }
return file.notes.size
val toImport = if (!skipExisting) {
file.notes
} else {
val existing = repository.observeAllIncludingTrashed().first()
.map { keyFor(it.createdAt, it.updatedAt, it.content) }
.toSet()
file.notes.filterNot { keyFor(it.createdAt, it.updatedAt, it.content) in existing }
}
val skipped = file.notes.size - toImport.size
BackupCodec.importSnapshot(file.copy(notes = toImport), repository) { cur, tot ->
onProgress(cur, tot)
}
return ImportSummary(imported = toImport.size, skipped = skipped)
}
/**
@@ -152,8 +174,26 @@ object BackupBridge {
?: throw IllegalArgumentException("Encrypted backup requires a passphrase")
return EncryptedBackupCodec.decrypt(bytes, pass)
}
/**
* M2 task 3 dedupe key: `(createdAt, updatedAt, sha256(content))`.
* Every component survives export/import on both platforms, and the
* timestamps are only trustworthy for dedupe because the repository
* now honours caller-supplied values (M2 import-timestamp fix).
* NUL-separated so distinct field splits can never collide; the
* content rides as a SHA-256 digest to keep the compared set cheap.
*/
private fun keyFor(createdAt: Long, updatedAt: Long, content: String): String {
val digest = java.security.MessageDigest.getInstance("SHA-256")
.digest(content.toByteArray(Charsets.UTF_8))
val hex = digest.joinToString("") { "%02x".format(it) }
return "$createdAt\u0000$updatedAt\u0000$hex"
}
}
/** Outcome of [BackupBridge.import] — what a duplicate-safe import did. */
data class ImportSummary(val imported: Int, val skipped: Int)
/** Lightweight DTO shown in the import confirmation dialog. */
data class BackupPreview(
val noteCount: Int,
@@ -74,7 +74,7 @@ fun main(args: Array<String>) {
step("import into fresh JVM library") {
ObjectBoxStore.init(libDir)
val n = runBlocking { BackupBridge.import(repo(), FileInputStream(src)) }
val n = runBlocking { BackupBridge.import(repo(), FileInputStream(src)).imported }
require(n == original.notes.size) { "imported $n, expected ${original.notes.size}" }
val inStore = runBlocking { repo().observeAllIncludingTrashed().first().size }
require(inStore == original.notes.size) { "store holds $inStore" }
@@ -110,7 +110,7 @@ fun main(args: Array<String>) {
ObjectBoxStore.reset()
val dir2 = File(libDir.absolutePath + "-enc").also { it.deleteRecursively(); it.mkdirs() }
ObjectBoxStore.init(dir2)
val n = runBlocking { BackupBridge.import(repo(), ByteArrayInputStream(envelope), pw()) }
val n = runBlocking { BackupBridge.import(repo(), ByteArrayInputStream(envelope), pw()).imported }
require(n == original.notes.size) { "imported $n, expected ${original.notes.size}" }
val withAtt = runBlocking { repo().observeAllIncludingTrashed().first() }
.first { it.attachments.isNotEmpty() }
@@ -121,6 +121,21 @@ fun main(args: Array<String>) {
require(bytes.contentEquals(expected)) { "attachment bytes differ (${bytes.size} vs ${expected.size})" }
"imported $n notes; attachment ${bytes.size}B byte-identical to the phone's"
}
step("dedupe: re-import same file") {
ObjectBoxStore.reset()
val dir3 = File(libDir.absolutePath + "-dedupe").also { it.deleteRecursively(); it.mkdirs() }
ObjectBoxStore.init(dir3)
val first = runBlocking { BackupBridge.import(repo(), FileInputStream(src)) }
require(first.imported == original.notes.size && first.skipped == 0) { "first=$first" }
val again = runBlocking { BackupBridge.import(repo(), FileInputStream(src)) }
require(again.imported == 0 && again.skipped == original.notes.size) { "again=$again" }
// The override must still behave like the old additive import.
val forced = runBlocking { BackupBridge.import(repo(), FileInputStream(src), skipExisting = false) }
require(forced.imported == original.notes.size) { "forced=$forced" }
val count = runBlocking { repo().observeAllIncludingTrashed().first().size }
require(count == original.notes.size * 2) { "store holds $count after forced re-import" }
"re-import skipped=${again.skipped}; override added ${forced.imported} (store=$count)"
}
step("close libraries") {
ObjectBoxStore.reset()
"clean"
@@ -2,9 +2,11 @@ package org.incido.desktop
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.material.icons.Icons
import androidx.compose.material.icons.filled.MoreVert
import androidx.compose.material3.AlertDialog
import androidx.compose.material3.Checkbox
import androidx.compose.material3.DropdownMenu
import androidx.compose.material3.DropdownMenuItem
import androidx.compose.material3.Icon
@@ -20,6 +22,7 @@ import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.text.input.PasswordVisualTransformation
import androidx.compose.ui.unit.dp
@@ -66,6 +69,8 @@ fun BackupActions(
var menuOpen by remember { mutableStateOf(false) }
var importStage by remember { mutableStateOf<ImportStage?>(null) }
var exportStage by remember { mutableStateOf<ExportStage?>(null) }
// M2 task 3: dedupe is ON by default; the confirm dialog exposes it.
var skipDuplicates by remember { mutableStateOf(true) }
fun say(message: String) { scope.launch { snackbar.showSnackbar(message) } }
@@ -131,6 +136,8 @@ fun BackupActions(
is ImportStage.Confirm -> ConfirmImportDialog(
preview = stage.preview,
skipExisting = skipDuplicates,
onSkipChange = { skipDuplicates = it },
onDismiss = { importStage = null },
onConfirm = {
importStage = null
@@ -141,9 +148,15 @@ fun BackupActions(
repository,
FileInputStream(stage.file),
stage.passphrase?.toCharArray(),
skipExisting = skipDuplicates,
)
}
}.onSuccess {
say(
"Imported ${it.imported} notes" +
if (it.skipped > 0) " · skipped ${it.skipped} already imported" else "",
)
}
}.onSuccess { say("Imported $it notes") }
.onFailure { say("Import failed: ${it.message}") }
}
},
@@ -225,6 +238,8 @@ private suspend fun startImport(
@Composable
private fun ConfirmImportDialog(
preview: BackupPreview,
skipExisting: Boolean,
onSkipChange: (Boolean) -> Unit,
onDismiss: () -> Unit,
onConfirm: () -> Unit,
) {
@@ -239,13 +254,14 @@ private fun ConfirmImportDialog(
"Exported by Incido ${preview.appVersion} on $exported",
style = MaterialTheme.typography.bodySmall,
)
Row(verticalAlignment = Alignment.CenterVertically) {
Checkbox(checked = skipExisting, onCheckedChange = onSkipChange)
Text(
"Existing notes are kept. (Re-importing the same file still duplicates — " +
"dedupe is M2 task 3.)",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.outline,
"Skip notes that are already here",
style = MaterialTheme.typography.bodyMedium,
)
}
}
},
confirmButton = { TextButton(onClick = onConfirm) { Text("Import") } },
dismissButton = { TextButton(onClick = onDismiss) { Text("Cancel") } },
@@ -228,13 +228,20 @@ private fun NotesWindow(
"BACKUP preview notes=${preview?.noteCount} " +
"attachments=${preview?.attachmentCount} app=${preview?.appVersion}",
)
val imported = BackupBridge.import(repository, FileInputStream(file))
println("BACKUP imported=$imported (same calls as the Import… menu)")
val first = BackupBridge.import(repository, FileInputStream(file))
println(
"BACKUP imported=${first.imported} skipped=${first.skipped} " +
"(same calls as the Import… menu)",
)
if (backupExport != null) {
val out = File(backupExport)
val exported = BackupBridge.exportPlain(repository, FileOutputStream(out), DESKTOP_APP_VERSION)
println("BACKUP exported=$exported notes to $out")
}
// M2 task 3: importing the same file again must add nothing —
// the dedupe path the confirm dialog's checkbox controls.
val again = BackupBridge.import(repository, FileInputStream(file))
println("BACKUP reimport imported=${again.imported} skipped=${again.skipped} (dedupe ON)")
}
}
if (exitAfterSeconds != null) {
+32
View File
@@ -370,5 +370,37 @@ list already has an exit-time AWT/`exitProcess` quirk).
Full suite green after the change: `assembleDebug`, `:core:data-jvm:smoke`,
`backupSmoke`, `:desktopApp:run --self-test`.
**M2 task 3 — import dedupe: DONE.** `BackupBridge.import` now skips notes
whose `(createdAt, updatedAt, sha256(content))` key already exists in the
library (default ON, `skipExisting = false` overrides), and returns
`ImportSummary(imported, skipped)` instead of a bare count. The existing
keys are snapshotted BEFORE the import, so two genuinely identical notes
inside one file still import into a fresh library — only a second import
of the same file adds nothing (matches the §4 wording exactly). Keys
include trashed notes, so restoring around the trash can't duplicate.
`BackupManager.import` (Android shim) passes the flag through and returns
the summary; `SettingsViewModel`'s snackbar gained the skip count (one
line — the screen layouts are untouched). The desktop confirm dialog
replaced its "duplicates persist" warning with a working
"Skip notes that are already here" checkbox (default ON); the export
format did not change (`formatVersion` stays 1).
```
backupSmoke new step: re-import skipped=27; override added 27 (store=54) PASS
:desktopApp:run --backup-import <real export> --backup-export … --exit-after 8
BACKUP imported=27 skipped=0 → exported=27 → reimport imported=0 skipped=27
```
Known limitation (by design, per plan): attachments are not part of the
key — two notes with identical text AND timestamps but different
attachments dedupe as one. Phone side: second import now says
"Imported 0 note(s), skipped 27 already imported." — device smoke still
pending with the rest of task 4.
Full suite green: `backupSmoke` (9 steps incl. dedupe), `assembleDebug`,
`:core:data-jvm:smoke`, scripted desktop run (clean exit),
`:desktopApp:run --self-test`.